Client closed before SASL negotiation finished.

[Gobi2511]

I cannot able to connect to spark from livy. When making request to create a session the session becomes dead with the following error.

Versions:
OS: macOS Sonoma 14.6.1
Python: 3.11.9
Spark: 3.5.4
Scala: 2.12
Livy: 0.8.0

Error:

stderr: 25/02/11 20:01:19 WARN Utils: Your hostname, MacBook-Pro-7.local resolves to a loopback address: 127.0.0.1; using 192.168.0.100 instead (on interface en0) 25/02/11 20:01:19 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 25/02/11 20:01:19 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 25/02/11 20:01:19 INFO RSCDriver: Connecting to: 192.168.0.100:10000 25/02/11 20:01:19 INFO RSCDriver: Starting RPC server... 25/02/11 20:01:20 INFO RpcServer: Connected to the port 10001 25/02/11 20:01:20 WARN ClientConf: Your hostname, MacBook-Pro-7.local, resolves to a loopback address; using 192.168.0.100 instead (on interface en0) 25/02/11 20:01:20 WARN ClientConf: Set 'livy.rsc.rpc.server.address' if you need to bind to another address. Exception in thread "main" java.util.concurrent.ExecutionException: javax.security.sasl.SaslException: Client closed before SASL negotiation finished. at io.netty.util.concurrent.DefaultPromise.get(DefaultPromise.java:351) at org.apache.livy.rsc.driver.RSCDriver.initializeServer(RSCDriver.java:203) at org.apache.livy.rsc.driver.RSCDriver.run(RSCDriver.java:336) at org.apache.livy.rsc.driver.RSCDriverBootstrapper.main(RSCDriverBootstrapper.java:93) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:569) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1034) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:199) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:222) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1125) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1134) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: javax.security.sasl.SaslException: Client closed before SASL negotiation finished. at org.apache.livy.rsc.rpc.Rpc$SaslClientHandler.dispose(Rpc.java:592) at org.apache.livy.rsc.rpc.SaslHandler.channelInactive(SaslHandler.java:92) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:305) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274) at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:411) at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:376) at io.netty.handler.codec.ByteToMessageCodec.channelInactive(ByteToMessageCodec.java:118) at org.apache.livy.rsc.rpc.KryoMessageCodec.channelInactive(KryoMessageCodec.java:97) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:813) at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.base/java.lang.Thread.run(Thread.java:840) 25/02/11 20:01:20 INFO ShutdownHookManager: Shutdown hook called 25/02/11 20:01:20 INFO ShutdownHookManager: Deleting directory /private/var/folders/xc/s8y7cw_d4sld08rn88b4_7480000gp/T/spark-7f7a4d73-f5c0-45e1-9f44-bbfbceecaa1e

Below are my configurations:

spark-default.conf:

## spark-default.conf:

spark.master                     local[*]
spark.serializer                 org.apache.spark.serializer.KryoSerializer

spark.default.parallelism	 4
spark.driver.memory              5g
spark.driver.cores		 4
spark.driver.memory 		 6g
spark.driver.host	192.168.0.100

spark.executor.cores 		 4
spark.executor.memory 		 6g
spark.executor.instances 	 6

spark.sql.adaptive.enabled	 true
spark.sql.adaptive.join.enabled	 true
spark.sql.caseSensitive		 true
spark.sql.crossJoin.enabled	 true
spark.shuffle.service.enabled 	 true
spark.dynamicAllocation.enabled  true


spark.authenticate	false
spark.authenticate.enableSasl	false
spark.yarn.security.tokens.enabled	false
spark.security.credentials.hdfs.enabled	false
spark.network.timeout	600s

Livy.conf

## Livy.conf

livy.rsc.auth.enabled=false
livy.rsc.rpc.server.address=192.168.0.100
livy.rsc.launcher.port.range=11000~11100
livy.server.session.timeout=600000
livy.server.rpc.timeout=600000

[cashokkumar]

@Gobi2511 we me too facing the same issues , tried multiple ways to fix this issue but all the ways / solutions failed

[Gobi2511]

This issues is happening especially when we try to make request to create a statement execution it is failing. Batches are working fine. Since there is not proper response from this repo, I'm going to use batches instead of statements.

[ns1000]

I am getting the same problem.

25/02/22 06:56:23 INFO RpcServer$SaslServerHandler: Exception in SASL negotiation. io.netty.handler.codec.DecoderException: java.lang.IllegalArgumentException: Unable to create serializer "com.esotericsoftware.kryo.serializers.FieldSerializer" for class: java.lang.invoke.SerializedLambda at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) at io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:103) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: java.lang.IllegalArgumentException: Unable to create serializer "com.esotericsoftware.kryo.serializers.FieldSerializer" for class: java.lang.invoke.SerializedLambda at com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:65) at com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:43) at com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:396) at com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:380) at com.esotericsoftware.kryo.Kryo.register(Kryo.java:410) at org.apache.livy.client.common.Serializer$1.initialValue(Serializer.java:55) at org.apache.livy.client.common.Serializer$1.initialValue(Serializer.java:44) at java.base/java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:195) at java.base/java.lang.ThreadLocal.get(ThreadLocal.java:172) at org.apache.livy.client.common.Serializer.deserialize(Serializer.java:67) at org.apache.livy.rsc.rpc.KryoMessageCodec.decode(KryoMessageCodec.java:70) at io.netty.handler.codec.ByteToMessageCodec$1.decode(ByteToMessageCodec.java:42) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ... 17 more Caused by: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481) at com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:51) ... 30 more Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field private final java.lang.Class java.lang.invoke.SerializedLambda.capturingClass accessible: module java.base does not "opens java.lang.invoke" to unnamed module @1e96a13e at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354) at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297) at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178) at java.base/java.lang.reflect.Field.setAccessible(Field.java:172) at com.esotericsoftware.kryo.serializers.FieldSerializer.buildValidFields(FieldSerializer.java:283) at com.esotericsoftware.kryo.serializers.FieldSerializer.rebuildCachedFields(FieldSerializer.java:216) at com.esotericsoftware.kryo.serializers.FieldSerializer.rebuildCachedFields(FieldSerializer.java:157) at com.esotericsoftware.kryo.serializers.FieldSerializer.<init>(FieldSerializer.java:150) at com.esotericsoftware.kryo.serializers.FieldSerializer.<init>(FieldSerializer.java:134) at com.esotericsoftware.kryo.serializers.FieldSerializer.<init>(FieldSerializer.java:130) ... 36 more 25/02/22 06:56:23 WARN DefaultChannelPipeline: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. io.netty.handler.codec.DecoderException: java.lang.IllegalArgumentException: Unable to create serializer "com.esotericsoftware.kryo.serializers.FieldSerializer" for class: java.lang.invoke.SerializedLambda at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) at io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:103) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: java.lang.IllegalArgumentException: Unable to create serializer "com.esotericsoftware.kryo.serializers.FieldSerializer" for class: java.lang.invoke.SerializedLambda at com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:65) at com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:43) at com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:396) at com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:380) at com.esotericsoftware.kryo.Kryo.register(Kryo.java:410) at org.apache.livy.client.common.Serializer$1.initialValue(Serializer.java:55) at org.apache.livy.client.common.Serializer$1.initialValue(Serializer.java:44) at java.base/java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:195) at java.base/java.lang.ThreadLocal.get(ThreadLocal.java:172) at org.apache.livy.client.common.Serializer.deserialize(Serializer.java:67) at org.apache.livy.rsc.rpc.KryoMessageCodec.decode(KryoMessageCodec.java:70) at io.netty.handler.codec.ByteToMessageCodec$1.decode(ByteToMessageCodec.java:42) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ... 17 more Caused by: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481) at com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:51) ... 30 more Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field private final java.lang.Class java.lang.invoke.SerializedLambda.capturingClass accessible: module java.base does not "opens java.lang.invoke" to unnamed module @1e96a13e at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354) at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297) at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178) at java.base/java.lang.reflect.Field.setAccessible(Field.java:172) at com.esotericsoftware.kryo.serializers.FieldSerializer.buildValidFields(FieldSerializer.java:283) at com.esotericsoftware.kryo.serializers.FieldSerializer.rebuildCachedFields(FieldSerializer.java:216) at com.esotericsoftware.kryo.serializers.FieldSerializer.rebuildCachedFields(FieldSerializer.java:157) at com.esotericsoftware.kryo.serializers.FieldSerializer.<init>(FieldSerializer.java:150) at com.esotericsoftware.kryo.serializers.FieldSerializer.<init>(FieldSerializer.java:134) at com.esotericsoftware.kryo.serializers.FieldSerializer.<init>(FieldSerializer.java:130) ... 36 more 25/02/22 06:56:23 INFO LineBufferedStream: Exception in thread "main" java.util.concurrent.ExecutionException: javax.security.sasl.SaslException: Client closed before SASL negotiation finished. 25/02/22 06:56:23 INFO LineBufferedStream: at io.netty.util.concurrent.DefaultPromise.get(DefaultPromise.java:351) 25/02/22 06:56:23 INFO LineBufferedStream: at org.apache.livy.rsc.driver.RSCDriver.initializeServer(RSCDriver.java:203) 25/02/22 06:56:23 INFO LineBufferedStream: at org.apache.livy.rsc.driver.RSCDriver.run(RSCDriver.java:336) 25/02/22 06:56:23 INFO LineBufferedStream: at org.apache.livy.rsc.driver.RSCDriverBootstrapper.main(RSCDriverBootstrapper.java:93) 25/02/22 06:56:23 INFO LineBufferedStream: at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 25/02/22 06:56:23 INFO LineBufferedStream: at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) 25/02/22 06:56:23 INFO LineBufferedStream: at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 25/02/22 06:56:23 INFO LineBufferedStream: at java.base/java.lang.reflect.Method.invoke(Method.java:569) 25/02/22 06:56:23 INFO LineBufferedStream: at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) 25/02/22 06:56:23 INFO LineBufferedStream: at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1029) 25/02/22 06:56:23 INFO LineBufferedStream: at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194) 25/02/22 06:56:23 INFO LineBufferedStream: at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217) 25/02/22 06:56:23 INFO LineBufferedStream: at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91) 25/02/22 06:56:23 INFO LineBufferedStream: at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120) 25/02/22 06:56:23 INFO LineBufferedStream: at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129) 25/02/22 06:56:23 INFO LineBufferedStream: at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 25/02/22 06:56:23 INFO LineBufferedStream: Caused by: javax.security.sasl.SaslException: Client closed before SASL negotiation finished. 25/02/22 06:56:23 INFO LineBufferedStream: at org.apache.livy.rsc.rpc.Rpc$SaslClientHandler.dispose(Rpc.java:592) 25/02/22 06:56:23 INFO LineBufferedStream: at org.apache.livy.rsc.rpc.SaslHandler.channelInactive(SaslHandler.java:92) 25/02/22 06:56:23 INFO LineBufferedStream: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:305) 25/02/22 06:56:23 INFO LineBufferedStream: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) 25/02/22 06:56:23 INFO LineBufferedStream: at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274) 25/02/22 06:56:23 INFO LineBufferedStream: at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:411) 25/02/22 06:56:23 INFO LineBufferedStream: at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:376) 25/02/22 06:56:23 INFO LineBufferedStream: at io.netty.handler.codec.ByteToMessageCodec.channelInactive(ByteToMessageCodec.java:118) 25/02/22 06:56:23 INFO LineBufferedStream: at org.apache.livy.rsc.rpc.KryoMessageCodec.channelInactive(KryoMessageCodec.java:97) 25/02/22 06:56:23 INFO LineBufferedStream: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303) 25/02/22 06:56:23 INFO LineBufferedStream: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) 25/02/22 06:56:23 INFO LineBufferedStream: at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274) 25/02/22 06:56:23 INFO LineBufferedStream: at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) 25/02/22 06:56:23 INFO LineBufferedStream: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301) 25/02/22 06:56:23 INFO LineBufferedStream: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) 25/02/22 06:56:23 INFO LineBufferedStream: at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) 25/02/22 06:56:23 INFO LineBufferedStream: at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:813) 25/02/22 06:56:23 INFO LineBufferedStream: at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) 25/02/22 06:56:23 INFO LineBufferedStream: at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) 25/02/22 06:56:23 INFO LineBufferedStream: at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) 25/02/22 06:56:23 INFO LineBufferedStream: at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566) 25/02/22 06:56:23 INFO LineBufferedStream: at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) 25/02/22 06:56:23 INFO LineBufferedStream: at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) 25/02/22 06:56:23 INFO LineBufferedStream: at java.base/java.lang.Thread.run(Thread.java:840) 25/02/22 06:56:23 INFO LineBufferedStream: 25/02/22 06:56:23 INFO ShutdownHookManager: Shutdown hook called 25/02/22 06:56:23 INFO LineBufferedStream: 25/02/22 06:56:23 INFO ShutdownHookManager: Deleting directory /tmp/spark-dbb2a28a-287b-4107-821e-8a1bbcdc20b3

[Gobi2511]

@ns1000 Try with batches instead of statements. It is working for my use case. Hope it would help for you as well.

[DevId-E]

Same here, hopefully there will be a fix anytime. Do you know of an alternative mantained project @Gobi2511 ?

[DevId-E]

I made some tests

yarn/client:

• spark: session get's created, spark, scala example 1+1 computes sucessfully
• pyspark: session get's created, interpreter failed to start

yarn/cluster:

• spark: session get's shut down
• pyspark: session get's shut down

using batches seems to work, but I only tested it with a simple hello world example

[fei819]

double check your jdk environment, probably set to 17. as livy was build under jdk 8, set jdk to 11 works for us.

[DevId-E]

Sorry for the late reply, i checked and every image is using jdk 11 on my side. @fei819 what spark version are you using? 3.5.3 and 3.5.4 are not working with sessions as it seems.

[NissalNodes]

@1e96a13e Unsupported java version