python - sklearn pipeline fit: AttributeError: lower not found -
i use pipeline in sklearn, this:
corpus = load_files('corpus/train') stop_words = [x x in open('stopwords.txt', 'r').read().split('\n')] # uppercase! countvec = countvectorizer(stop_words=stop_words, ngram_range=(1, 2)) x_train, x_test, y_train, y_test = train_test_split(corpus.data, corpus.target, test_size=0.9, random_state=0) x_train_counts = countvec.fit_transform(x_train) x_test_counts = countvec.transform(x_test) k_fold = kfold(n=len(corpus.data), n_folds=6) confusion = np.array([[0, 0], [0, 0]]) pipeline = pipeline([ ('vectorizer', countvectorizer(stop_words=stop_words, ngram_range=(1, 2))), ('classifier', multinomialnb()) ]) train_indices, test_indices in k_fold: pipeline.fit(x_train_counts, y_train) predictions = pipeline.predict(x_test_counts)
however, getting error:
attributeerror: lower not found
i have looked @ post:
attributeerror: lower not found; using pipeline countvectorizer in scikit-learn
but passing list of bytes vectorizer, shouldnt problem.
edit
corpus = load_files('corpus') stop_words = [x x in open('stopwords.txt', 'r').read().split('\n')] x_train, x_test, y_train, y_test = train_test_split(corpus.data, corpus.target, test_size=0.5, random_state=0) k_fold = kfold(n=len(corpus.data), n_folds=6) confusion = np.array([[0, 0], [0, 0]]) pipeline = pipeline([ ('vectorizer', countvectorizer(stop_words=stop_words, ngram_range=(1, 2))), ('classifier', multinomialnb())]) train_indices, test_indices in k_fold: pipeline.fit(x_train[train_indices], y_train[train_indices]) predictions = pipeline.predict(x_test[test_indices])
now error:
typeerror: integer arrays 1 element can converted index
2nd edit
corpus = load_files('corpus') stop_words = [y x in open('stopwords.txt', 'r').read().split('\n') y in (x, x.title())] k_fold = kfold(n=len(corpus.data), n_folds=6) confusion = np.array([[0, 0], [0, 0]]) pipeline = pipeline([ ('vectorizer', countvectorizer(stop_words=stop_words, ngram_range=(1, 2))), ('classifier', multinomialnb())]) train_indices, test_indices in k_fold: pipeline.fit(corpus.data, corpus.target)
you not using correctly pipeline. don't need pass data vectorized, idea pipeline vectorizes data.
# done pipeline # x_train_counts = countvec.fit_transform(x_train) # x_test_counts = countvec.transform(x_test) k_fold = kfold(n=len(corpus.data), n_folds=6) confusion = np.array([[0, 0], [0, 0]]) pipeline = pipeline([ ('vectorizer', countvectorizer(stop_words=stop_words, ngram_range=(1, 2))), ('classifier', multinomialnb()) ]) # not using indices... train_indices, test_indices in k_fold: pipeline.fit(corpus.data[train_indices], corpus.target[train_indices]) predictions = pipeline.predict(corpus.data[test_indices])
Comments
Post a Comment