Spark Streaming REST Custom Receiver -


is possible use rest api in custom receiver spark streaming?

i trying able multiple calls / reads api asynchronously , use spark streaming it.

a custom receiver can whatever process produces data asynchronously. typically, def receive() method send async requests rest server, maybe using using futures , dedicated threadpool. oncompletion of future, call store(data) method give results spark streaming job. in nutshell,

  • def onstart() => creates process manages async request response handling
  • def receive() => continuously i/o , reports results through calling store(...)
  • def onstop() => stops process , cleans onstart creates.

there's example in custom receivers docs.


Comments

Popular posts from this blog

aws api gateway - SerializationException in posting new Records via Dynamodb Proxy Service in API -

asp.net - Problems sending emails from forum -