Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.4.0
-
Reviewed
Description
Problem description
I start hadoop cluster and timeservice based on hdfs store ,it throw execption when accessing http://timelineserver:8088/ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo&userid=hadoop&flowrunid=1692343022285, and detailed logs includes exception:
2023-08-18 17:40:51 2023-08-18 09:40:51 INFO TimelineReaderWebServices:3372 - Processed URL /ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo&userid=hadoop&flowrunid=1&clusterid=yjd&appid=application_1692341890246_0001 (Took 56 ms.) 2023-08-18 17:40:51 2023-08-18 09:40:51 WARN GenericExceptionHandler:96 - INTERNAL_SERVER_ERROR 2023-08-18 17:40:51 javax.ws.rs.WebApplicationException: java.io.FileNotFoundException: File /tmp/hadoop-hadoop/timeline_service_data/entities/yjd/hadoop/QuasiMonteCarlo/1/application_1692341890246_0001 does not exist. 2023-08-18 17:40:51 at org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.handleException(TimelineReaderWebServices.java:200) 2023-08-18 17:40:51 at org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntityTypes(TimelineReaderWebServices.java:3368) 2023-08-18 17:40:51 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2023-08-18 17:40:51 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 2023-08-18 17:40:51 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2023-08-18 17:40:51 at java.lang.reflect.Method.invoke(Method.java:498) 2023-08-18 17:40:51 at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) 2023-08-18 17:40:51 at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) 2023-08-18 17:40:51 at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) 2023-08-18 17:40:51 at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) 2023-08-18 17:40:51 at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) 2023-08-18 17:40:51 at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) 2023-08-18 17:40:51 at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) 2023-08-18 17:40:51 at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) 2023-08-18 17:40:51 at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) 2023-08-18 17:40:51 at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) 2023-08-18 17:40:51 at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) 2023-08-18 17:40:51 at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) 2023-08-18 17:40:51 at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) 2023-08-18 17:40:51 at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) 2023-08-18 17:40:51 at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) 2023-08-18 17:40:51 at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) 2023-08-18 17:40:51 at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) 2023-08-18 17:40:51 at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656) 2023-08-18 17:40:51 at org.apache.hadoop.yarn.server.timelineservice.reader.security.TimelineReaderWhitelistAuthorizationFilter.doFilter(TimelineReaderWhitelistAuthorizationFilter.java:85) 2023-08-18 17:40:51 at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) 2023-08-18 17:40:51 at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) 2023-08-18 17:40:51 at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:650) 2023-08-18 17:40:51 at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592) 2023-08-18 17:40:51 at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) 2023-08-18 17:40:51 at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) 2023-08-18 17:40:51 at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:110) 2023-08-18 17:40:51 at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) 2023-08-18 17:40:51 at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) 2023-08-18 17:40:51 at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1794) 2023-08-18 17:40:51 at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) 2023-08-18 17:40:51 at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) 2023-08-18 17:40:51 at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) 2023-08-18 17:40:51 at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) 2023-08-18 17:40:51 at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) 2023-08-18 17:40:51 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) 2023-08-18 17:40:51 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) 2023-08-18 17:40:51 at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) 2023-08-18 17:40:51 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 2023-08-18 17:40:51 at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) 2023-08-18 17:40:51 at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) 2023-08-18 17:40:51 at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) 2023-08-18 17:40:51 at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) 2023-08-18 17:40:51 at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) 2023-08-18 17:40:51 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) 2023-08-18 17:40:51 at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) 2023-08-18 17:40:51 at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) 2023-08-18 17:40:51 at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) 2023-08-18 17:40:51 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) 2023-08-18 17:40:51 at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) 2023-08-18 17:40:51 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 2023-08-18 17:40:51 at org.eclipse.jetty.server.Server.handle(Server.java:516) 2023-08-18 17:40:51 at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) 2023-08-18 17:40:51 at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) 2023-08-18 17:40:51 at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) 2023-08-18 17:40:51 at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) 2023-08-18 17:40:51 at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) 2023-08-18 17:40:51 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) 2023-08-18 17:40:51 at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) 2023-08-18 17:40:51 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) 2023-08-18 17:40:51 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) 2023-08-18 17:40:51 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) 2023-08-18 17:40:51 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) 2023-08-18 17:40:51 at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) 2023-08-18 17:40:51 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) 2023-08-18 17:40:51 at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) 2023-08-18 17:40:51 at java.lang.Thread.run(Thread.java:748) 2023-08-18 17:40:51 Caused by: java.io.FileNotFoundException: File /tmp/hadoop-hadoop/timeline_service_data/entities/yjd/hadoop/QuasiMonteCarlo/1/application_1692341890246_0001 does not exist. 2023-08-18 17:40:51 at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:1117) 2023-08-18 17:40:51 at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:149) 2023-08-18 17:40:51 at org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1188) 2023-08-18 17:40:51 at org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1185) 2023-08-18 17:40:51 at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) 2023-08-18 17:40:51 at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:1195) 2023-08-18 17:40:51 at org.apache.hadoop.yarn.server.timelineservice.storage.FileSystemTimelineReaderImpl.getEntityTypes(FileSystemTimelineReaderImpl.java:442) 2023-08-18 17:40:51 at org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderManager.getEntityTypes(TimelineReaderManager.java:209) 2023-08-18 17:40:51 at org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntityTypes(TimelineReaderWebServices.java:3364) 2023-08-18 17:40:51 ... 70 more
Problem analysis
The entity type write path of application is ${yarn.timeline-service.fs-writer.root-dir}/entities/${clusterId}/${userId}/${flowName}/${flowVersion}/${flowRun}/${appId}/${entityType} ,
but read path does not include ${flowRun} part.
This is the cause of FileNotFoundException
Problem solution
changing flowRun of read path into '*', and the read path is ${yarn.timeline-service.fs-writer.root-dir}/entities/${clusterId}/${userId}/${flowName}/${flowVersion}/*/${appId}/${entityType}
Attachments
Issue Links
- links to