python - sklearn pipeline fit: AttributeError: lower not found -


i use pipeline in sklearn, this:

corpus = load_files('corpus/train')  stop_words = [x x in open('stopwords.txt', 'r').read().split('\n')]  # uppercase!  countvec = countvectorizer(stop_words=stop_words, ngram_range=(1, 2))  x_train, x_test, y_train, y_test = train_test_split(corpus.data, corpus.target, test_size=0.9,                                                     random_state=0) x_train_counts = countvec.fit_transform(x_train) x_test_counts = countvec.transform(x_test)  k_fold = kfold(n=len(corpus.data), n_folds=6) confusion = np.array([[0, 0], [0, 0]])  pipeline = pipeline([     ('vectorizer',  countvectorizer(stop_words=stop_words, ngram_range=(1, 2))),     ('classifier',  multinomialnb()) ])  train_indices, test_indices in k_fold:      pipeline.fit(x_train_counts, y_train)     predictions = pipeline.predict(x_test_counts) 

however, getting error:

attributeerror: lower not found 

i have looked @ post:

attributeerror: lower not found; using pipeline countvectorizer in scikit-learn

but passing list of bytes vectorizer, shouldnt problem.

edit

corpus = load_files('corpus')  stop_words = [x x in open('stopwords.txt', 'r').read().split('\n')]  x_train, x_test, y_train, y_test = train_test_split(corpus.data, corpus.target, test_size=0.5,                                                     random_state=0)  k_fold = kfold(n=len(corpus.data), n_folds=6) confusion = np.array([[0, 0], [0, 0]])  pipeline = pipeline([     ('vectorizer', countvectorizer(stop_words=stop_words, ngram_range=(1, 2))),     ('classifier', multinomialnb())])  train_indices, test_indices in k_fold:     pipeline.fit(x_train[train_indices], y_train[train_indices])     predictions = pipeline.predict(x_test[test_indices]) 

now error:

typeerror: integer arrays 1 element can converted index 

2nd edit

corpus = load_files('corpus')  stop_words = [y x in open('stopwords.txt', 'r').read().split('\n') y in (x, x.title())]  k_fold = kfold(n=len(corpus.data), n_folds=6) confusion = np.array([[0, 0], [0, 0]])  pipeline = pipeline([     ('vectorizer', countvectorizer(stop_words=stop_words, ngram_range=(1, 2))),     ('classifier', multinomialnb())])  train_indices, test_indices in k_fold:     pipeline.fit(corpus.data, corpus.target) 

you not using correctly pipeline. don't need pass data vectorized, idea pipeline vectorizes data.

# done pipeline # x_train_counts = countvec.fit_transform(x_train) # x_test_counts = countvec.transform(x_test)  k_fold = kfold(n=len(corpus.data), n_folds=6) confusion = np.array([[0, 0], [0, 0]])  pipeline = pipeline([     ('vectorizer',  countvectorizer(stop_words=stop_words, ngram_range=(1, 2))),     ('classifier',  multinomialnb()) ])  # not using indices... train_indices, test_indices in k_fold:      pipeline.fit(corpus.data[train_indices], corpus.target[train_indices])     predictions = pipeline.predict(corpus.data[test_indices]) 

Comments

Popular posts from this blog

aws api gateway - SerializationException in posting new Records via Dynamodb Proxy Service in API -

asp.net - Problems sending emails from forum -