问题一:flink-1.11.2 job启动不起来怎么办?
启动命令:run -d -m yarn-cluster -p 12 -yjm 1600 -ytm 12288 -ys 12 -ynm xxx -yqu flink-critical -j /app/flink-1.11.2/executor/fcbox-streaming-sql-platform-1.11.jar --sqlid 17 --jobName realtime_app_kpi_dis_day_16
12个并行度, 12个slot,启动不了
Caused by: java.util.concurrent.CompletionException: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: No pooled slot available and request to ResourceManager for new slot failed at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ... 29 common frames omitted Caused by: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: No pooled slot available and request to ResourceManager for new slot failed ... 27 common frames omitted Caused by: org.apache.flink.runtime.resourcemanager.exceptions.ResourceManagerException: Could not fulfill slot request d0d4eb5366379663f826d9cbf1eb3ff0. at org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl.registerSlotRequest(SlotManagerImpl.java:391) at org.apache.flink.runtime.resourcemanager.ResourceManager.requestSlot(ResourceManager.java:457) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:284) at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:199) ... 20 common frames omitted Caused by: org.apache.flink.runtime.resourcemanager.exceptions.UnfulfillableSlotRequestException: Could not fulfill slot request d0d4eb5366379663f826d9cbf1eb3ff0. Requested resource profile (ResourceProfile{UNKNOWN}) is unfulfillable at org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl.lambda$fulfillPendingSlotRequestWithPendingTaskManagerSlot$11(SlotManagerImpl.java:882) at org.apache.flink.util.OptionalConsumer.ifNotPresent(OptionalConsumer.java:52) at org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl.fulfillPendingSlotRequestWithPendingTaskManagerSlot(SlotManagerImpl.java:878) at org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl.lambda$internalRequestSlot$9(SlotManagerImpl.java:865) at org.apache.flink.util.OptionalConsumer.ifNotPresent(OptionalConsumer.java:52) at org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl.internalRequestSlot(SlotManagerImpl.java:865) at org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl.registerSlotRequest(SlotManagerImpl.java:386) ... 27 common frames omitted *来自志愿者整理的flink邮件归档
参考答案:
你好,此问题我也遇到。目前已解决。
解决办法:
查看你pom文件中是否引入了hadoop相关依赖,Flink 1.11需要把hadoop相关依赖注释掉。*来自志愿者整理的flink邮件归档
关于本问题的更多回答可点击进行查看:
https://developer.aliyun.com/ask/370102?spm=a2c6h.13066369.question.68.33bf585fdxAytC
问题二:flink-1.11.2提交到yarn一直处于CREATED中
flink-1.11.2提交到yarn一直处于CREATED中,不会运行,flink-1.11.1没问题 资源已经分配
*来自志愿者整理的flink邮件归档
参考答案:
会不会是这个问题 https://issues.apache.org/jira/browse/FLINK-19151
*来自志愿者整理的flink邮件归档
关于本问题的更多回答可点击进行查看:
https://developer.aliyun.com/ask/370105?spm=a2c6h.13066369.question.67.33bf585fNybasw
问题三:FlinkSQL导致Prometheus内存暴涨
FlinkSQL 生成的Metrics数据 task_name名字超长,导致Prometheus查询的时候内存暴涨,生产环境接受不了。
下面只是一个最简单的例子,复杂的SQL生成的task_name长达9000字节。这会导致Prometheus内存暴涨,我该怎么办。
task_name="Source:_wuren_foo_ods_foo____SourceConversion_table__Unregistered_DataStream_1___fields__id__name______SinkConversionToRow____SourceConversion_table__default_catalog_default_database_ods_foo___fields__id__name__PROCTIME______Calc_select__id__name______SinkConversionToTuple2____Sink:_Unnamed"
*来自志愿者整理的flink邮件归档
参考答案:
是的,个人觉得可以提供一个配置项来控制 task Name。 完整的 task name 有助于排查问题等,简短的 task name 有助于在生产环境中 metric 的采集,可以极大较少发送的网络开销,存储空间等。 已建立个了 issue :https://issues.apache.org/jira/browse/FLINK-20375
*来自志愿者整理的flink邮件归档
关于本问题的更多回答可点击进行查看:
https://developer.aliyun.com/ask/370104?spm=a2c6h.13066369.question.70.33bf585fX97fbX
问题四:flink-json 函数用法
尝试使用flink-sql将聚合结果json展示的时候发现flink是支持JSON_OBJECTAGG, JSON_ARRAY, JSON_OBJECT 等这种函数的(使用的默认的blink), 但是总是报错函数的用法不对,有相关资料来介绍这些函数的使用方法的吗?或者示例
*来自志愿者整理的flink邮件归档
参考答案:
目前Flink SQL应该还没有正式支持json函数吧,上面的报错信息看起来也是符合预期的,说的是目前还找不到这个函数。
相关信息可以参考:https://issues.apache.org/jira/browse/FLINK-9477
*来自志愿者整理的flink邮件归档
关于本问题的更多回答可点击进行查看:
https://developer.aliyun.com/ask/370036?spm=a2c6h.13066369.question.69.33bf585fE2l6pr
问题五:Flink SQL 是否存在类似MySQL的group_concat函数
不知Flink SQL中是否存在类似MySQL的group_concat函数的功能: 例如: 数据: +------+-------+ | Id | Name | +------+-------+ | 10 | Larry | | 11 | Mike | | 12 | John | | 10 | Elon | | 10 | Bob | | 11 | Sam | +------+———+ 执行SQL :select Id,group_concat(Name SEPARATOR ',') as resultName from test group by Id; 期望得到的结果: +------+-----------------+ | Id | resultName | +------+-----------------+ | 10 | Larry,Elon,Bob | | 11 | Mike,Sam | | 12 | John | +------+————————+
这种在统计最近一段时间用户的登录IP,登录的设备等场景会用到,不知道目前的Flink SQL是否有类似的函数?
*来自志愿者整理的flink邮件归档
参考答案:
I think you are looking for LISTAGG [1] which is more SQL standard
compliant.
*来自志愿者整理的flink邮件归档
关于本问题的更多回答可点击进行查看:
https://developer.aliyun.com/ask/370035?spm=a2c6h.13066369.question.72.33bf585f2RMLdn