Get elements of type structure of row by name in SPARK SCALA -
in dataframe object in apache spark (i'm using scala interface), if i'm iterating on row objects, there way extract structure values name?
i using below code extract name facing problem on how read struct value .
if values had been of type string have done this:
 val resultdf=joineddf.rdd.map{row=>        val id=row.getas[long]("id")       val values=row.getas[string]("slotsize")       val feilds=row.getas[string](values)       (id,values,feilds)       }.todf("id","values","feilds")   but in case values has below schema
v1: struct (nullable = true)      |    |-- level1: string (nullable = true)      |    |-- level2: string (nullable = true)      |    |-- level3: string (nullable = true)      |    |-- level4: string (nullable = true)      |    |-- level5: string (nullable = true)   what shall replace line make code work given value has above structure.
  row.getas[string](values)      
you can access struct elements first extracting row (structs modeled row in spark) toplevel row this:
val level1 = row.getas[row]("struct").getas[string]("level1")      
Comments
Post a Comment