Spark Streaming REST Custom Receiver -
is possible use rest api in custom receiver spark streaming?
i trying able multiple calls / reads api asynchronously , use spark streaming it.
a custom receiver can whatever process produces data asynchronously. typically, def receive()
method send async requests rest server, maybe using using futures
, dedicated threadpool
. oncompletion
of future, call store(data)
method give results spark streaming job. in nutshell,
def onstart()
=> creates process manages async request response handlingdef receive()
=> continuously i/o , reports results through callingstore(...)
def onstop()
=> stops process , cleansonstart
creates.
there's example in custom receivers docs.
Comments
Post a Comment