lexRankr/0000755000175100001440000000000013056034162012056 5ustar hornikuserslexRankr/inst/0000755000175100001440000000000013056025704013035 5ustar hornikuserslexRankr/inst/doc/0000755000175100001440000000000013056025704013602 5ustar hornikuserslexRankr/inst/doc/Analyzing_Twitter_with_LexRankr.html.asis0000644000175100001440000000021713056025704023745 0ustar hornikusers%\VignetteIndexEntry{Analyzing Twitter with LexRankr} %\VignetteEngine{R.rsp::asis} %\VignetteKeyword{twitter} %\VignetteKeyword{lexrankr} lexRankr/inst/doc/Analyzing_Twitter_with_LexRankr.html0000644000175100001440000005444713056025704023025 0ustar hornikusers Using lexRankr to find a user’s most representative tweets

Using lexRankr to find a user’s most representative tweets

Adam Spannbauer

2017-03-01

Packages Used

library(lexRankr)
library(tidyverse)
library(stringr)
library(httr)
library(jsonlite)

In this document we get tweets from twitter using the twitter API and then analyze the tweets using lexRankr in order to find a user’s most representative tweets. If you don’t care about interacting with the twitter api you can jump to the lexrank analysis.

Get user tweets

Before we can analyze tweets we’ll need some tweets to analyze. We’ll be using Twitter’s API, and you’ll need to set up an account to get all keys needed for the api. The credentials needed for the api are: consumer key, consumer secret, token, and token secret. Below is how to set up your credentials to use the twitter api in this vignette.

# set api tokens/keys/secrets as environment vars
# Sys.setenv(cons_key     = 'my_cons_key')
# Sys.setenv(cons_secret  = 'my_cons_sec')
# Sys.setenv(token        = 'my_token')
# Sys.setenv(token_secret = 'my_token_sec')

#sign oauth
auth <- httr::oauth_app("twitter", key=Sys.getenv("cons_key"), secret=Sys.getenv("cons_secret"))
sig  <- httr::sign_oauth1.0(auth, token=Sys.getenv("token"), token_secret=Sys.getenv("token_secret"))

Now that we have our credentials set up, let’s write a function to get a user’s tweets from the api. Below the function get_timeline_df is defined. The function takes a user’s twitter handle, the number of tweets to get from the api, and the credentials we just set up. The function will return a dataframe with the columns created_at, favorite_count, retweet_count, text. The twitter api limits 200 tweets per get, so we will use a loop until we get the desired number of tweets.

get_timeline_df <- function(user, n_tweets=200, oauth_sig) {
  i <- 0
  n_left <- n_tweets
  timeline_df <- NULL
  #loop until n_tweets are all got
  while (n_left > 0) {
    n_to_get <- min(200, n_left)
    i <- i+1
    #incorporae max id in get_url (so as not to download same 200 tweets repeatedly)
    if (i==1) {
      get_url <- paste0("https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=",
                       user,"&count=", n_to_get)
    } else {
      get_url <- paste0("https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=",
                       user,"&count=",n_to_get,"&max_id=", max_id)
    }
    #GET tweets
    response <- httr::GET(get_url, oauth_sig)
    #extract content and clean up
    response_content <- httr::content(response)
    json_content     <- jsonlite::toJSON(response_content)
    #clean out evil special chars
    json_conv <- iconv(json_content, "UTF-8", "ASCII", sub = "") %>%
      stringr::str_replace_all("\003", "") #special character (^C) not caught by above clean
    timeline_list <- jsonlite::fromJSON(json_conv)
    #extract desired fields
    fields_i_care_about <- c("id", "text", "favorite_count", "retweet_count", "created_at")
    timeline_df <- purrr::map(fields_i_care_about, ~unlist(timeline_list[[.x]])) %>% 
      purrr::set_names(fields_i_care_about) %>% 
      dplyr::as_data_frame() %>% 
      dplyr::bind_rows(timeline_df) %>% 
      dplyr::distinct()
    #store min id (oldest tweet) to set as max id for next GET
    max_id <- min(purrr::map_dbl(timeline_list$id, 1))
    #update number of tweets left
    n_left <- n_left-n_to_get
  }
  return(timeline_df)
}

We can now use our function to gather a user’s tweets with the additional information of date-time, favorites, retweets. Lets use one of the most famous twitter accounts as of late: [@realDonaldTrump](https://twitter.com/realDonaldTrump).

tweets_df <- get_timeline_df("realDonaldTrump", 600, sig) %>% 
    mutate(text = str_replace_all(text, "\n", " ")) #clean out newlines for display

tweets_df %>% 
  head(n=3) %>% 
  select(text, created_at) %>% 
  knitr::kable()
text created_at
Yes, it is true - Carlos Slim, the great businessman from Mexico, called me about getting together for a meeting. We met, HE IS A GREAT GUY! Tue Dec 20 20:27:57 +0000 2016
especially how to get people, even with an unlimited budget, out to vote in the vital swing states ( and more). They focused on wrong states Tue Dec 20 13:09:18 +0000 2016
Bill Clinton stated that I called him after the election. Wrong, he called me (with a very nice congratulations). He “doesn’t know much” … Tue Dec 20 13:03:59 +0000 2016

Lexrank Analysis

We now have a dataframe that contains a column of tweets. This column of tweets will be the subject of the rest of the analysis. With the data in this format, we only need to call the bind_lexrank function to apply the lexrank algorithm to the tweets. The function will add a column of lexrank scores. The higher the lexrank score the more representative the tweet is of the tweets that we downloaded.

note: typically one would parse documents into sentences before applying lexrank (?unnest_sentences); however we will equate tweets to sentences for this analysis

tweets_df %>% 
  bind_lexrank(text, id, level="sentences") %>% 
  arrange(desc(lexrank)) %>% 
  head(n=5) %>% 
  select(text, lexrank) %>% 
  knitr::kable(caption = "Most Representative @realDonaldTrump Tweets")
Most Representative @realDonaldTrump Tweets
text lexrank
MAKE AMERICA GREAT AGAIN! 0.0087551
Well, the New Year begins. We will, together, MAKE AMERICA GREAT AGAIN! 0.0085258
HAPPY PRESIDENTS DAY - MAKE AMERICA GREAT AGAIN! 0.0082361
Happy Thanksgiving to everyone. We will, together, MAKE AMERICA GREAT AGAIN! 0.0060486
Hopefully, all supporters, and those who want to MAKE AMERICA GREAT AGAIN, will go to D.C. on January 20th. It will be a GREAT SHOW! 0.0059713

Repeating tweetRank analysis for other users

With our get_timeline_df function we can easily repeat this analysis for other users. Below we repeat the whole analysis in a single magrittr pipeline.

get_timeline_df("dog_rates", 600, sig) %>% 
  mutate(text = str_replace_all(text, "\n", " ")) %>% 
  bind_lexrank(text, id, level="sentences") %>% 
  arrange(desc(lexrank)) %>% 
  head(n=5) %>% 
  select(text, lexrank) %>% 
  knitr::kable(caption = "Most Representative @dog_rates Tweets")
Most Representative @dog_rates Tweets
text lexrank
@Lin_Manuel good day good dog 0.0167123
Please keep loving 0.0099864
Here we h*ckin go 0.0085708
Last day to get anything from our Valentine’s Collection by Valentine’s Day! Shop: https://t.co/MXljGLH3qY https://t.co/qFBCMytKMB 0.0077583
Even if I tried (which I would never), I’d last like 17 seconds 0.0073899



lexRankr/tests/0000755000175100001440000000000013056025671013225 5ustar hornikuserslexRankr/tests/testthat.R0000644000175100001440000000010013056025671015177 0ustar hornikuserslibrary(testthat) library(lexRankr) test_check("lexRankr") lexRankr/tests/testthat/0000755000175100001440000000000013056034162015060 5ustar hornikuserslexRankr/tests/testthat/test-unnest_sentences.R0000644000175100001440000000461413056025671021555 0ustar hornikuserscontext("unnest_sentences") # test output str -------------------------------------------------------- test_that("correct ouput class and str", { df <- data.frame(doc_id = 1:3, text = c("Testing the system. Second sentence for you.", "System testing the tidy documents df.", "Documents will be parsed and lexranked."), stringsAsFactors = FALSE) test_result <- unnest_sentences(df, out, text) expect_equal(dim(test_result), c(4,3)) expect_true(is.data.frame(test_result)) expect_equal(names(test_result), c("doc_id","sent_id","out")) test_result <- unnest_sentences(df, out, text, drop=FALSE) expect_equal(dim(test_result), c(4,4)) expect_equal(names(test_result), c("doc_id","text","sent_id","out")) }) # test bad input ------------------------------------------------------- test_that("test input checking", { df <- data.frame(doc_id = 1:3, text = c("Testing the system. Second sentence for you.", "System testing the tidy documents df.", "Documents will be parsed and lexranked."), stringsAsFactors = FALSE) expect_error(unnest_sentences(df, out, fake)) expect_error(unnest_sentences(NULL, out, text)) expect_error(unnest_sentences(df, out, text, drop = NULL)) }) # test output val ------------------------------------------------------ test_that("output value", { df <- data.frame(doc_id = 1:3, text = c("Testing the system. Second sentence for you.", "System testing the tidy documents df.", "Documents will be parsed and lexranked."), stringsAsFactors = FALSE) test_result <- unnest_sentences(df, out, text) expected_result <- data.frame(doc_id = c(1L, 1L, 2L, 3L), sent_id = c(1L, 2L, 1L, 1L), out = c("Testing the system.", "Second sentence for you.", "System testing the tidy documents df.", "Documents will be parsed and lexranked."), stringsAsFactors = FALSE) expect_equal(test_result, expected_result) }) lexRankr/tests/testthat/test-lexRankFromSimil.R0000644000175100001440000000633213056025671021417 0ustar hornikuserscontext("lexRankFromSimil") # test object out str and class --------------------------------------- test_that("object out str and class", { testDocs <- c("Testing 1, 2, 3.", "Is everything working as expected in my test?", "Is it working?") tokenDf <- sentenceTokenParse(testDocs)$tokens similDf <- sentenceSimil(sentenceId = tokenDf$sentenceId, token = tokenDf$token, docId = tokenDf$docId) testResult <- lexRankFromSimil(similDf$sent1, similDf$sent2, similDf$similVal) expect_equal(class(testResult), "data.frame") expect_equal(names(testResult), c("sentenceId", "value")) expect_true(is.character(testResult$sentenceId)) expect_true(is.numeric(testResult$value)) }) # test bad inputs --------------------------------------- test_that("bad inputs", { testDocs <- c("Testing 1, 2, 3.", "Is everything working as expected in my test?", "Is it working?") tokenDf <- sentenceTokenParse(testDocs)$tokens similDf <- sentenceSimil(sentenceId = tokenDf$sentenceId, token = tokenDf$token, docId = tokenDf$docId) expect_error(lexRankFromSimil(NULL, similDf$sent2, similDf$similVal)) expect_error(lexRankFromSimil(c(1,2), similDf$sent2, similDf$similVal)) expect_error(lexRankFromSimil(similDf$sent1, similDf$sent2, c("a","b","c"))) expect_error(lexRankFromSimil(similDf$sent1, similDf$sent2, similDf$similVal, threshold = NULL)) expect_error(lexRankFromSimil(similDf$sent1, similDf$sent2, similDf$similVal, damping = NULL)) }) # test object out value test_that("object out value", { testDocs <- c("Testing 1, 2, 3.", "Is everything working as expected in my test?", "Is it working?") tokenDf <- sentenceTokenParse(testDocs)$tokens similDf <- sentenceSimil(sentenceId = tokenDf$sentenceId, token = tokenDf$token, docId = tokenDf$docId) testResult <- lexRankFromSimil(similDf$sent1, similDf$sent2, similDf$similVal) %>% dplyr::mutate(value = round(value, 5)) expectedResult <- data.frame(sentenceId = c("1_1", "2_1", "3_1"), value = c(0.25676, 0.48649, 0.25676), stringsAsFactors = FALSE) expect_identical(testResult, expectedResult) testResult <- lexRankFromSimil(similDf$sent1, similDf$sent2, similDf$similVal, continuous = TRUE) %>% dplyr::mutate(value = round(value, 5)) expectedResult <- data.frame(sentenceId = c("1_1", "2_1", "3_1"), value = c(0.25676, 0.48649, 0.25676), stringsAsFactors = FALSE) expect_identical(testResult, expectedResult) testResult <- lexRankFromSimil(similDf$sent1, similDf$sent2, similDf$similVal, usePageRank = FALSE) %>% dplyr::mutate(value = round(value, 5)) expectedResult <- data.frame(sentenceId = c("2_1", "1_1", "3_1"), value = c(2, 1, 1), stringsAsFactors = FALSE) expect_identical(testResult, expectedResult) }) lexRankr/tests/testthat/test-idfCosine.R0000644000175100001440000000274113056025671020074 0ustar hornikuserscontext("lexRankr:::idfCosineSimil") # test bad inputs --------------------------------------- test_that("bad inputs to idf cosine", { expect_error(lexRankr:::idfCosineSimil(NULL)) badMat <- matrix(c("a","b","c","d"), nrow=2) expect_error(lexRankr:::idfCosineSimil(badMat)) }) # test object out str and class --------------------------------------- test_that("object out str and class", { testMat <- matrix(runif(9, min = .01, max = 1), nrow=3) testResult <- lexRankr:::idfCosineSimil(testMat) expect_equal(class(testResult), "numeric") expect_equal(length(testResult), 3) }) # test object out value test_that("object out value", { testMat <- matrix(c(1,0,0,0,1,0,0,0,1), nrow=3) expect_equal(lexRankr:::idfCosineSimil(testMat), c(0,0,0)) testMat <- matrix(c(0,0,0,0,0,0,0,0,0), nrow=3) expect_equal(lexRankr:::idfCosineSimil(testMat), c(NaN,NaN,NaN)) testMat <- matrix(c(1,1,1,1,1,1,1,1,1), nrow=3) expect_equal(lexRankr:::idfCosineSimil(testMat), c(1,1,1)) testMat <- matrix(runif(9, min = .01, max = 1), nrow=3) rcppIdf <- round(lexRankr:::idfCosineSimil(testMat), 10) #pure r version comparison idfCosine <- function(x,y) { res <- sum(x*y)/(sqrt(sum(x^2))*sqrt(sum(y^2))) return(round(res, 10)) } elem1 <- idfCosine(testMat[1,], testMat[2,]) elem2 <- idfCosine(testMat[1,], testMat[3,]) elem3 <- idfCosine(testMat[2,], testMat[3,]) expect_equal(rcppIdf, c(elem1, elem2, elem3)) }) lexRankr/tests/testthat/test-sentenceTokenParse.R0000644000175100001440000000304113056025671021763 0ustar hornikuserscontext("sentenceTokenParse") # test output classes ---------------------------------------- test_that("object class and structure check", { testDocs <- c("12345", "Testing 1, 2, 3.", "Is everything working as expected Mr. Wickham?") testResult <- sentenceTokenParse(testDocs) expect_equal(class(testResult), "list") expect_equal(unique(vapply(testResult, class, character(1))), "data.frame") expect_equal(names(testResult$tokens), c("docId","sentenceId","token")) expect_true(is.numeric(testResult$tokens$docId)) expect_true(is.character(testResult$tokens$sentenceId)) expect_true(is.character(testResult$tokens$sentence)) }) # test output value ------------------------------------------- test_that("All clean options TRUE", { testDocs <- c("Testing 1, 2, 3.", "Is everything working as expected Mr. Wickham?") testResult <- sentenceTokenParse(testDocs, docId = "create", removePunc=TRUE, removeNum=TRUE, toLower=TRUE, stemWords=TRUE, rmStopWords=TRUE) expectedResultSentences <- sentenceParse(testDocs) expectedResultTokens <- lexRankr::tokenize(testDocs) %>% unlist() %>% .[which(!is.na(.))] expect_equal(testResult$sentences, expectedResultSentences) expect_equal(testResult$tokens$token, expectedResultTokens) expect_equal(class(testResult), "list") }) lexRankr/tests/testthat/test-sentenceSimil.R0000644000175100001440000000466313056025671021000 0ustar hornikuserscontext("sentenceSimil") # test object out str and class --------------------------------------- test_that("testing result str and class", { testDocs <- c("Testing 1, 2, 3.", "Is everything working as expected in my test?", "Is it working?") tokenDf <- sentenceTokenParse(testDocs)$tokens testResult <- sentenceSimil(sentenceId = tokenDf$sentenceId, token = tokenDf$token, docId = tokenDf$docId) expect_equal(class(testResult), "data.frame") expect_equal(names(testResult), c("sent1","sent2","similVal")) expect_true(is.character(testResult$sent1)) expect_true(is.character(testResult$sent2)) expect_true(is.numeric(testResult$similVal)) }) test_that("bad input", { expect_error(sentenceSimil(sentenceId = c("1_1"), token = c("word","word2"), docId = c(1,2))) expect_error(sentenceSimil(sentenceId = c("1_1", "2_1"), token = c(1,2), docId = c(1,2))) testDocs <- c("test","test") tokenDf <- sentenceTokenParse(testDocs)$tokens expect_error(sentenceSimil(sentenceId = tokenDf$sentenceId, token = tokenDf$token, docId = tokenDf$docId)) testDocs <- c("1","2") tokenDf <- sentenceTokenParse(testDocs)$tokens expect_error(sentenceSimil(sentenceId = tokenDf$sentenceId, token = tokenDf$token, docId = tokenDf$docId)) }) # test output value --------------------------------------- test_that("output value check", { testDocs <- c("Testing 1, 2, 3.", "Is everything working as expected in my test?", "Is it working?") tokenDf <- sentenceTokenParse(testDocs)$tokens testResult <- sentenceSimil(sentenceId = tokenDf$sentenceId, token = tokenDf$token, docId = tokenDf$docId) %>% dplyr::mutate(similVal = round(similVal, 5)) expectedResult <- data.frame(sent1 = c("1_1", "1_1", "2_1"), sent2 = c("2_1", "3_1", "3_1"), similVal = c(0.32718, 0, 0.32718), stringsAsFactors = FALSE) expect_equal(testResult, expectedResult) }) lexRankr/tests/testthat/test-unnest_sentences_.R0000644000175100001440000000476613056025671021724 0ustar hornikuserscontext("unnest_sentences_") # test output str -------------------------------------------------------- test_that("correct ouput class and str", { df <- data.frame(doc_id = 1:3, text = c("Testing the system. Second sentence for you.", "System testing the tidy documents df.", "Documents will be parsed and lexranked."), stringsAsFactors = FALSE) test_result <- unnest_sentences_(df, "out", "text") expect_equal(dim(test_result), c(4,3)) expect_true(is.data.frame(test_result)) expect_equal(names(test_result), c("doc_id","sent_id","out")) test_result <- unnest_sentences_(df, "out", "text", drop=FALSE) expect_equal(dim(test_result), c(4,4)) expect_equal(names(test_result), c("doc_id","text","sent_id","out")) }) # test bad input ------------------------------------------------------- test_that("test input checking", { df <- data.frame(doc_id = 1:3, text = c("Testing the system. Second sentence for you.", "System testing the tidy documents df.", "Documents will be parsed and lexranked."), stringsAsFactors = FALSE) expect_error(unnest_sentences_(df, "out", "fake")) expect_error(unnest_sentences_(NULL, "out", "text")) expect_error(unnest_sentences_(df, "out", "text", drop = NULL)) expect_warning(unnest_sentences_(df, "out", "text", c("test","test2"))) }) # test output val ------------------------------------------------------ test_that("output value", { df <- data.frame(doc_id = 1:3, text = c("Testing the system. Second sentence for you.", "System testing the tidy documents df.", "Documents will be parsed and lexranked."), stringsAsFactors = FALSE) test_result <- unnest_sentences_(df, "out", "text") expected_result <- data.frame(doc_id = c(1L, 1L, 2L, 3L), sent_id = c(1L, 2L, 1L, 1L), out = c("Testing the system.", "Second sentence for you.", "System testing the tidy documents df.", "Documents will be parsed and lexranked."), stringsAsFactors = FALSE) expect_equal(test_result, expected_result) }) lexRankr/tests/testthat/test-tokenize.R0000644000175100001440000001205713056025671020022 0ustar hornikuserscontext("tokenize") # test tokenize output classes ---------------------------------------- test_that("All clean options TRUE", { testDocs <- c("12345", "Testing 1, 2, 3.", "Is everything working as expected Mr. Wickham?") testResult <- tokenize(testDocs) expect_equal(class(testResult), "list") expect_equal(unique(vapply(testResult, class, character(1))), "character") }) # test bad input ------------------------------------------------------- test_that("test input checking", { expect_error(tokenize(NULL)) expect_error(tokenize(data.frame(badInput="test"))) expect_error(tokenize("test", removePunc=NULL)) expect_error(tokenize("test", removeNum=NULL)) expect_error(tokenize("test", toLower=NULL)) expect_error(tokenize("test", stemWords=NULL)) expect_error(tokenize("test", rmStopWords=NULL)) }) # test tokenize and arg option variations ------------------------------ test_that("All clean options TRUE", { testDocs <- c("Testing 1, 2, 3.", "Is everything working as expected Mr. Wickham?") testResult <- tokenize(testDocs, removePunc=TRUE, removeNum=TRUE, toLower=TRUE, stemWords=TRUE, rmStopWords=TRUE) expectedResult <- list("test", c("work","expect","mr","wickham")) expect_equal(testResult, expectedResult) expect_equal(class(testResult), "list") }) test_that("All clean options FALSE", { testDocs <- c("Testing 1, 2, 3", "Is everything working as expected Mr. Wickham?") testResult <- tokenize(testDocs, removePunc=FALSE, removeNum=FALSE, toLower=FALSE, stemWords=FALSE, rmStopWords=FALSE) expectedResult <- list(c("Testing", "1", ",", "2", ",", "3"), c("Is", "everything", "working", "as", "expected", "Mr", ".", "Wickham", "?")) expect_equal(testResult, expectedResult) expect_equal(class(testResult), "list") }) test_that("Single option tests: removePunc = FALSE", { testDocs <- c("Testing 1, 2, 3.", "Is everything working as expected Mr. Wickham?") testResult <- tokenize(testDocs, removePunc=FALSE, removeNum=TRUE, toLower=TRUE, stemWords=TRUE, rmStopWords=TRUE) expectedResult <- list(c("test",",",",","." ), c("work","expect","mr",".","wickham","?" )) expect_equal(testResult, expectedResult) expect_equal(class(testResult), "list") }) test_that("Single option tests: removeNum = FALSE", { testDocs <- c("Testing 1, 2, 3", "Is everything working as expected Mr. Wickham?") testResult <- tokenize(testDocs, removePunc=TRUE, removeNum=FALSE, toLower=TRUE, stemWords=TRUE, rmStopWords=TRUE) expectedResult <- list(c("test","1","2","3"), c("work","expect","mr","wickham")) expect_equal(testResult, expectedResult) expect_equal(class(testResult), "list") }) test_that("Single option tests: toLower = FALSE", { testDocs <- c("Testing 1, 2, 3", "Is everything working as expected Mr. Wickham?") testResult <- tokenize(testDocs, removePunc=TRUE, removeNum=TRUE, toLower=FALSE, stemWords=TRUE, rmStopWords=TRUE) expectedResult <- list(c("Test"), c("work","expect","Mr","Wickham")) expect_equal(testResult, expectedResult) expect_equal(class(testResult), "list") }) test_that("Single option tests: stemWords = FALSE", { testDocs <- c("Testing 1, 2, 3", "Is everything working as expected Mr. Wickham?") testResult <- tokenize(testDocs, removePunc=TRUE, removeNum=TRUE, toLower=TRUE, stemWords=FALSE, rmStopWords=TRUE) expectedResult <- list(c("testing"), c("working","expected","mr","wickham")) expect_equal(testResult, expectedResult) expect_equal(class(testResult), "list") }) test_that("Single option tests: rmStopWords = FALSE", { testDocs <- c("Testing 1, 2, 3", "Is everything working as expected Mr. Wickham?") testResult <- tokenize(testDocs, removePunc=TRUE, removeNum=TRUE, toLower=TRUE, stemWords=TRUE, rmStopWords=FALSE) expectedResult <- list(c("test"), c("i","everyth","work","a", "expect", "mr", "wickham")) expect_equal(testResult, expectedResult) expect_equal(class(testResult), "list") }) lexRankr/tests/testthat/test-sentenceParse.R0000644000175100001440000000332013056025671020762 0ustar hornikuserscontext("sentenceParse") # test sentence object structure----------------------------------------------- test_that("sentenceParse output class and structure check", { testDoc <- "Testing one, two, three. Is everything working as expected Mr. Wickham?" testResult <- sentenceParse(testDoc) expect_equal(class(testResult), "data.frame") expect_equal(names(testResult), c("docId", "sentenceId", "sentence")) expect_true(is.numeric(testResult$docId)) expect_true(is.character(testResult$sentenceId)) expect_true(is.character(testResult$sentence)) }) # test bad input ------------------------------------------------------- test_that("test input checking", { expect_error(sentenceParse(NULL)) expect_error(sentenceParse(data.frame(badInput="test"))) expect_error(sentenceParse("test", docId = c("fake","fake2"))) expect_error(sentenceParse(c("test","test2"), docId = "fake")) expect_error(sentenceParse(c("test","test2"), docId = NULL)) }) # test sentence output value ----------------------------------------------- test_that("Example doc parses sentences as expected", { testDoc <- "Testing one, two, three. Is everything working as expected Mr. Wickham?" testResult <- sentenceParse(testDoc) expectedResult <- data.frame(docId = c(1L, 1L), sentenceId = c("1_1", "1_2"), sentence = c("Testing one, two, three.", "Is everything working as expected Mr. Wickham?"), stringsAsFactors = FALSE) expect_equal(testResult, expectedResult) expect_equal(class(testResult), "data.frame") expect_equal })lexRankr/tests/testthat/test-lexRank.R0000644000175100001440000000307313056025671017574 0ustar hornikuserscontext("lexRank") # test object out str and class --------------------------------------- test_that("object out str and class", { testDocs <- c("Testing 1, 2, 3.", "Is everything working as expected in my test?", "Is it working?") testResult <- lexRank(testDocs, Verbose = FALSE) expect_equal(class(testResult), "data.frame") expect_equal(names(testResult), c("docId","sentenceId", "sentence","value")) expect_true(is.character(testResult$sentenceId)) expect_true(is.character(testResult$sentence)) expect_true(is.numeric(testResult$value)) }) # test bad inputs --------------------------------------- test_that("bad inputs", { expect_error(lexRank(FALSE, Verbose = FALSE)) expect_error(lexRank(NULL, Verbose = FALSE)) }) # test object out value test_that("object out value", { testDocs <- c("Testing 1, 2, 3.", "Is everything working as expected in my test?", "Is it working?") testResult <- lexRank(testDocs, Verbose = FALSE) %>% dplyr::mutate(value = round(value, 5)) expectedResult <- data.frame(docId = c(2L, 1L, 3L), sentenceId = c("2_1", "1_1", "3_1"), sentence = c("Is everything working as expected in my test?", "Testing 1, 2, 3.", "Is it working?"), value = c(0.48649, 0.25676, 0.25676), stringsAsFactors = FALSE) expect_identical(testResult, expectedResult) }) lexRankr/tests/testthat/test-bind_lexrank_.R0000644000175100001440000002212013056025671020761 0ustar hornikuserscontext("bind_lexrank_") # test output str -------------------------------------------------------- test_that("correct ouput class and str", { df <- data.frame(doc_id = 1:3, text = c("Testing the system. Second sentence for you.", "System testing the tidy documents df.", "Documents will be parsed and lexranked."), stringsAsFactors = FALSE) test_result <- df %>% unnest_sentences(sents, text) %>% bind_lexrank_("sents", "doc_id", level = 'sentences') expect_equal(dim(test_result), c(4,4)) expect_true(is.data.frame(test_result)) expect_equal(names(test_result), c("doc_id","sent_id","sents","lexrank")) test_result <- df %>% unnest_sentences(sents, text, drop=FALSE) %>% bind_lexrank_("sents", "doc_id", level = 'sentences') expect_equal(dim(test_result), c(4,5)) expect_equal(names(test_result), c("doc_id","text","sent_id","sents","lexrank")) df <- data.frame(doc_id = c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 3L, 3L, 3L, 3L, 3L, 3L), sent_id = c(1L, 1L, 1L, 2L, 2L, 2L, 2L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L), sents = c("Testing the system.", "Testing the system.", "Testing the system.", "Second sentence for you.", "Second sentence for you.", "Second sentence for you.", "Second sentence for you.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked."), tokens = c("testing", "the", "system", "second", "sentence", "for", "you", "system", "testing", "the", "tidy", "documents", "df", "documents", "will", "be", "parsed", "and", "lexranked"), stringsAsFactors = FALSE) test_result <- df %>% bind_lexrank_("tokens", "doc_id", "sent_id", "tokens") expect_equal(dim(test_result), c(19,5)) expect_equal(names(test_result), c("doc_id","sent_id","sents","tokens","lexrank")) }) # test bad input ------------------------------------------------------- test_that("test input checking", { df <- data.frame(doc_id = 1:3, text = c("Testing the system. Second sentence for you.", "System testing the tidy documents df.", "Documents will be parsed and lexranked."), stringsAsFactors = FALSE) %>% unnest_sentences(sents, text) expect_error(bind_lexrank_(df, "sents", "fake")) expect_error(bind_lexrank_(NULL, "sents", "doc_id")) expect_error(bind_lexrank_(df, "sents", "doc_id", level="fake")) expect_warning(bind_lexrank_(df, "sents", "doc_id", level=c("sentences","tokens"))) df <- data.frame(doc_id = c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 3L, 3L, 3L, 3L, 3L, 3L), sent_id = c(1L, 1L, 1L, 2L, 2L, 2L, 2L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L), sents = c("Testing the system.", "Testing the system.", "Testing the system.", "Second sentence for you.", "Second sentence for you.", "Second sentence for you.", "Second sentence for you.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked."), tokens = c("testing", "the", "system", "second", "sentence", "for", "you", "system", "testing", "the", "tidy", "documents", "df", "documents", "will", "be", "parsed", "and", "lexranked"), stringsAsFactors = FALSE) expect_error(bind_lexrank_(df, "tokens", "doc_id", "fake", level="tokens")) expect_error(bind_lexrank_(df, "tokens", "doc_id", level="tokens")) expect_warning(bind_lexrank_(df, "tokens", "doc_id", "sent_id", level=c("tokens","sentences"))) }) # test output val ------------------------------------------------------ test_that("output value", { df <- data.frame(doc_id = 1:3, text = c("Testing the system. Second sentence for you.", "System testing the tidy documents df.", "Documents will be parsed and lexranked."), stringsAsFactors = FALSE) %>% unnest_sentences(sents, text) test_result <- bind_lexrank_(df, "sents", "doc_id", level="sentences") expected_result <- data.frame(doc_id = c(1L, 1L, 2L, 3L), sent_id = c(1L, 2L, 1L, 1L), sents = c("Testing the system.", "Second sentence for you.", "System testing the tidy documents df.", "Documents will be parsed and lexranked."), lexrank = c(0.5, NA, 0.5, NA), stringsAsFactors = FALSE) expect_equal(test_result, expected_result) df <- data.frame(doc_id = c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 3L, 3L, 3L, 3L, 3L, 3L), sent_id = c(1L, 1L, 1L, 2L, 2L, 2L, 2L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L), sents = c("Testing the system.", "Testing the system.", "Testing the system.", "Second sentence for you.", "Second sentence for you.", "Second sentence for you.", "Second sentence for you.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked."), tokens = c("testing", "the", "system", "second", "sentence", "for", "you", "system", "testing", "the", "tidy", "documents", "df", "documents", "will", "be", "parsed", "and", "lexranked"), stringsAsFactors = FALSE) test_result <- bind_lexrank_(df, "tokens", "doc_id", "sent_id", level="sentences") test_result$lexrank <- round(test_result$lexrank, 5) expected_result <- data.frame(doc_id = c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 3L, 3L, 3L, 3L, 3L, 3L), sent_id = c(1L, 1L, 1L, 2L, 2L, 2L, 2L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L), sents = c("Testing the system.", "Testing the system.", "Testing the system.", "Second sentence for you.", "Second sentence for you.", "Second sentence for you.", "Second sentence for you.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked."), tokens = c("testing", "the", "system", "second", "sentence", "for", "you", "system", "testing", "the", "tidy", "documents", "df", "documents", "will", "be", "parsed", "and", "lexranked"), lexrank = c(0.16667, NA, 0.16667, NA, NA, NA, NA, 0.16667, 0.16667, NA, NA, 0.16667, NA, 0.16667, NA, NA, NA, NA, NA), stringsAsFactors = FALSE) expect_equal(test_result, expected_result) }) lexRankr/tests/testthat/test-bind_lexrank.R0000644000175100001440000002200713056025671020626 0ustar hornikuserscontext("bind_lexrank") # test output str -------------------------------------------------------- test_that("correct ouput class and str", { df <- data.frame(doc_id = 1:3, text = c("Testing the system. Second sentence for you.", "System testing the tidy documents df.", "Documents will be parsed and lexranked."), stringsAsFactors = FALSE) test_result <- df %>% unnest_sentences(sents, text) %>% bind_lexrank(sents, doc_id, level = 'sentences') expect_equal(dim(test_result), c(4,4)) expect_true(is.data.frame(test_result)) expect_equal(names(test_result), c("doc_id","sent_id","sents","lexrank")) test_result <- df %>% unnest_sentences(sents, text, drop=FALSE) %>% bind_lexrank(sents,doc_id, level = 'sentences') expect_equal(dim(test_result), c(4,5)) expect_equal(names(test_result), c("doc_id","text","sent_id","sents","lexrank")) df <- data.frame(doc_id = c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 3L, 3L, 3L, 3L, 3L, 3L), sent_id = c(1L, 1L, 1L, 2L, 2L, 2L, 2L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L), sents = c("Testing the system.", "Testing the system.", "Testing the system.", "Second sentence for you.", "Second sentence for you.", "Second sentence for you.", "Second sentence for you.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked."), tokens = c("testing", "the", "system", "second", "sentence", "for", "you", "system", "testing", "the", "tidy", "documents", "df", "documents", "will", "be", "parsed", "and", "lexranked"), stringsAsFactors = FALSE) test_result <- df %>% bind_lexrank(tokens, doc_id, sent_id, "tokens") expect_equal(dim(test_result), c(19,5)) expect_equal(names(test_result), c("doc_id","sent_id","sents","tokens","lexrank")) }) # test bad input ------------------------------------------------------- test_that("test input checking", { df <- data.frame(doc_id = 1:3, text = c("Testing the system. Second sentence for you.", "System testing the tidy documents df.", "Documents will be parsed and lexranked."), stringsAsFactors = FALSE) %>% unnest_sentences(sents, text) expect_error(bind_lexrank(df, sents, fake)) expect_error(bind_lexrank(NULL, sents, doc_id)) expect_error(bind_lexrank(df, sents, doc_id, level="fake")) expect_warning(bind_lexrank(df, sents, doc_id, level=c("sentences","tokens"))) df <- data.frame(doc_id = c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 3L, 3L, 3L, 3L, 3L, 3L), sent_id = c(1L, 1L, 1L, 2L, 2L, 2L, 2L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L), sents = c("Testing the system.", "Testing the system.", "Testing the system.", "Second sentence for you.", "Second sentence for you.", "Second sentence for you.", "Second sentence for you.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked."), tokens = c("testing", "the", "system", "second", "sentence", "for", "you", "system", "testing", "the", "tidy", "documents", "df", "documents", "will", "be", "parsed", "and", "lexranked"), stringsAsFactors = FALSE) expect_error(bind_lexrank(df, tokens, doc_id, fake, level="tokens")) expect_error(bind_lexrank(df, tokens, doc_id, level="tokens")) expect_warning(bind_lexrank(df, tokens, doc_id, sent_id, level=c("tokens","sentences"))) }) # test output val ------------------------------------------------------ test_that("output value", { df <- data.frame(doc_id = 1:3, text = c("Testing the system. Second sentence for you.", "System testing the tidy documents df.", "Documents will be parsed and lexranked."), stringsAsFactors = FALSE) %>% unnest_sentences(sents, text) test_result <- bind_lexrank(df, sents, doc_id, level="sentences") expected_result <- data.frame(doc_id = c(1L, 1L, 2L, 3L), sent_id = c(1L, 2L, 1L, 1L), sents = c("Testing the system.", "Second sentence for you.", "System testing the tidy documents df.", "Documents will be parsed and lexranked."), lexrank = c(0.5, NA, 0.5, NA), stringsAsFactors = FALSE) expect_equal(test_result, expected_result) df <- data.frame(doc_id = c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 3L, 3L, 3L, 3L, 3L, 3L), sent_id = c(1L, 1L, 1L, 2L, 2L, 2L, 2L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L), sents = c("Testing the system.", "Testing the system.", "Testing the system.", "Second sentence for you.", "Second sentence for you.", "Second sentence for you.", "Second sentence for you.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked."), tokens = c("testing", "the", "system", "second", "sentence", "for", "you", "system", "testing", "the", "tidy", "documents", "df", "documents", "will", "be", "parsed", "and", "lexranked"), stringsAsFactors = FALSE) test_result <- bind_lexrank(df, tokens, doc_id, sent_id, level="sentences") test_result$lexrank <- round(test_result$lexrank, 5) expected_result <- data.frame(doc_id = c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 3L, 3L, 3L, 3L, 3L, 3L), sent_id = c(1L, 1L, 1L, 2L, 2L, 2L, 2L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L), sents = c("Testing the system.", "Testing the system.", "Testing the system.", "Second sentence for you.", "Second sentence for you.", "Second sentence for you.", "Second sentence for you.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "System testing the tidy documents df.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked.", "Documents will be parsed and lexranked."), tokens = c("testing", "the", "system", "second", "sentence", "for", "you", "system", "testing", "the", "tidy", "documents", "df", "documents", "will", "be", "parsed", "and", "lexranked"), lexrank = c(0.16667, NA, 0.16667, NA, NA, NA, NA, 0.16667, 0.16667, NA, NA, 0.16667, NA, 0.16667, NA, NA, NA, NA, NA), stringsAsFactors = FALSE) expect_equal(test_result, expected_result) }) lexRankr/src/0000755000175100001440000000000013056025705012650 5ustar hornikuserslexRankr/src/idfCosineSimil.cpp0000644000175100001440000000147313056025705016262 0ustar hornikusers#include using namespace Rcpp; double idfCosineSimilVector(NumericVector x, NumericVector y) { int n=x.size(); double numerator=0; double denomenatorX=0; double denomenatorY=0; double result; for (int i = 0; i #include #include // for NULL #include /* FIXME: Check these declarations against the C/Fortran source code. */ /* .Call calls */ extern SEXP lexRankr_idfCosineSimil(SEXP); static const R_CallMethodDef CallEntries[] = { {"lexRankr_idfCosineSimil", (DL_FUNC) &lexRankr_idfCosineSimil, 1}, {NULL, NULL, 0} }; void R_init_lexRankr(DllInfo *dll) { R_registerRoutines(dll, NULL, CallEntries, NULL, NULL); R_useDynamicSymbols(dll, FALSE); } lexRankr/src/RcppExports.cpp0000644000175100001440000000103713056025705015646 0ustar hornikusers// Generated by using Rcpp::compileAttributes() -> do not edit by hand // Generator token: 10BE3573-1514-4C36-9D1C-5A225CD40393 #include using namespace Rcpp; // idfCosineSimil NumericVector idfCosineSimil(NumericMatrix mat); RcppExport SEXP lexRankr_idfCosineSimil(SEXP matSEXP) { BEGIN_RCPP Rcpp::RObject rcpp_result_gen; Rcpp::RNGScope rcpp_rngScope_gen; Rcpp::traits::input_parameter< NumericMatrix >::type mat(matSEXP); rcpp_result_gen = Rcpp::wrap(idfCosineSimil(mat)); return rcpp_result_gen; END_RCPP } lexRankr/NAMESPACE0000644000175100001440000000057713056025671013313 0ustar hornikusers# Generated by roxygen2: do not edit by hand export(bind_lexrank) export(bind_lexrank_) export(lexRank) export(lexRankFromSimil) export(sentenceParse) export(sentenceSimil) export(sentenceTokenParse) export(tokenize) export(unnest_sentences) export(unnest_sentences_) importFrom(Rcpp,sourceCpp) importFrom(magrittr,"%>%") importFrom(utils,combn) useDynLib(lexRankr) lexRankr/NEWS.md0000644000175100001440000000167713056025671013174 0ustar hornikusers# lexRankr 0.4.0 * added functions `unnest_sentences` and `unnest_sentences_` to parse sentences in a dataframe following tidy data principles * added functions `bind_lexrank` and `bind_lexrank_` to calculate lexrank scores for sentences in a dataframe following tidy data principles (`unnest_sentences` & `bind_lexrank` can be used on a df in a magrittr pipeline) * added vignette for using lexrank to analyze tweets # lexRankr 0.3.0 * sentence similarity from `sentenceSimil` now calculated using Rcpp. Improves speed by ~25%-30% over old implementation using `proxy` package # lexRankr 0.2.0 * Added logic to avoid naming conflicts in proxy::pr_DB in `sentenceSimil` (#1, @AdamSpannbauer) * Added check and error for cases where no sentences above threshold in `lexRankFromSimil` (#2, @AdamSpannbauer) * `tokenize` now has stricter punctuation removal. Removes all non-alphnumeric characters as opposed to removing `[:punct:]` lexRankr/R/0000755000175100001440000000000013056025671012264 5ustar hornikuserslexRankr/R/sentenceTokenParse.R0000644000175100001440000000617713056025671016222 0ustar hornikusersutils::globalVariables(c("token")) #' Parse text into sentences and tokens #' @description Parse a character vector of documents into into both sentences and a clean vector of tokens. The resulting output includes IDs for document and sentence for use in other \code{lexRank} functions. #' @param text A character vector of documents to be parsed into sentences and tokenized. #' @param docId A character vector of document Ids the same length as \code{text}. If \code{docId=="create"} document Ids will be created. #' @param removePunc \code{TRUE} or \code{FALSE} indicating whether or not to remove punctuation from \code{text} while tokenizing. If \code{TRUE}, puncuation will be removed. Defaults to \code{TRUE}. #' @param removeNum \code{TRUE} or \code{FALSE} indicating whether or not to remove numbers from \code{text} while tokenizing. If \code{TRUE}, numbers will be removed. Defaults to \code{TRUE}. #' @param toLower \code{TRUE} or \code{FALSE} indicating whether or not to coerce all of \code{text} to lowercase while tokenizing. If \code{TRUE}, \code{text} will be coerced to lowercase. Defaults to \code{TRUE}. #' @param stemWords \code{TRUE} or \code{FALSE} indicating whether or not to stem resulting tokens. If \code{TRUE}, the outputted tokens will be tokenized using \code{SnowballC::wordStem()}. Defaults to \code{TRUE}. #' @param rmStopWords \code{TRUE}, \code{FALSE}, or character vector of stopwords to remove from tokens. If \code{TRUE}, words in \code{tm::stopwords("SMART")} will be removed prior to stemming. If \code{FALSE}, no stopword removal will occur. If a character vector is passed, this vector will be used as the list of stopwords to be removed. Defaults to \code{TRUE}. #' @return A list of dataframes. The first element of the list returned is the \code{sentences} dataframe; this dataframe has columns \code{docId}, \code{sentenceId}, & \code{sentence} (the actual text of the sentence). The second element of the list returned is the \code{tokens} dataframe; this dataframe has columns \code{docId}, \code{sentenceId}, & \code{token} (the actual text of the token). #' @examples #' sentenceTokenParse(c("Bill is trying to earn a Ph.D.", "You have to have a 5.0 GPA."), #' docId=c("d1","d2")) #' @importFrom magrittr "%>%" #' @export sentenceTokenParse <- function(text, docId = "create", removePunc=TRUE, removeNum=TRUE, toLower=TRUE, stemWords=TRUE, rmStopWords=TRUE){ sentenceDf <- sentenceParse(text, docId=docId) tokenDfList <- lapply(seq_along(sentenceDf$sentence), function(i) { sentVec <- sentenceDf$sentence[i] tokenList <- tokenize(text = sentVec, removePunc = removePunc, removeNum = removeNum, toLower = toLower, stemWords = stemWords, rmStopWords=rmStopWords) subTokenDfList <- lapply(seq_along(tokenList), function(j) { data.frame(docId=sentenceDf$docId[i], sentenceId=sentenceDf$sentenceId[i], token=tokenList[[j]], stringsAsFactors = FALSE) }) dplyr::bind_rows(subTokenDfList) }) tokenDf <- dplyr::bind_rows(tokenDfList) %>% dplyr::filter(!is.na(token)) class(tokenDf) <- "data.frame" list(sentences=sentenceDf, tokens=tokenDf) } lexRankr/R/bind_lexrank.R0000644000175100001440000001454613056025671015061 0ustar hornikusers#' Bind lexrank scores to a dataframe of text #' @description Bind lexrank scores to a dataframe of sentences or to a dataframe of tokens with sentence ids #' @param tbl dataframe containing column of sentences to be lexranked #' @param text name of column containing sentences or tokens to be lexranked #' @param doc_id name of column containing document ids corresponding to \code{text} #' @param sent_id Only needed if \code{level} is "tokens". name of column containing sentence ids corresponding to \code{text} #' @param level the parsed level of the text column to be lexranked. i.e. is \code{text} a column of "sentences" or "tokens"? The "tokens" level is provided to allow users to implement custom tokenization. Note: even if the input \code{level} is "tokens" lexrank scores are assigned at the sentence level. #' @param threshold The minimum simililarity value a sentence pair must have to be represented in the graph where lexRank is calculated. #' @param usePageRank \code{TRUE} or \code{FALSE} indicating whether or not to use the page rank algorithm for ranking sentences. If \code{FALSE}, a sentences unweighted centrality will be used as the rank. Defaults to \code{TRUE}. #' @param damping The damping factor to be passed to page rank algorithm. Ignored if \code{usePageRank} is \code{FALSE}. #' @param continuous \code{TRUE} or \code{FALSE} indicating whether or not to use continuous LexRank. Only applies if \code{usePageRank==TRUE}. If \code{TRUE}, \code{threshold} will be ignored and lexRank will be computed using a weighted graph representation of the sentences. Defaults to \code{FALSE}. #' @param ... tokenizing options to be passed to lexRankr::tokenize. Ignored if \code{level} is "sentences" #' @return A dataframe with an additional column of lexrank scores (column is given name lexrank) #' @examples #' library(dplyr) #' #' df <- dplyr::tibble(doc_id = 1:3, #' text = c("Testing the system. Second sentence for you.", #' "System testing the tidy documents df.", #' "Documents will be parsed and lexranked.")) #' #' df %>% #' unnest_sentences(sents, text) %>% #' bind_lexrank(sents, doc_id, level = 'sentences') #' #' df %>% #' unnest_sentences(sents, text) %>% #' bind_lexrank_("sents", "doc_id", level = 'sentences') #' #' df <- data.frame(doc_id = c(1, 1, 1, 1, 1, 1, 1, 2, 2, 2, #' 2, 2, 2, 3, 3, 3, 3, 3, 3), #' sent_id = c(1, 1, 1, 2, 2, 2, 2, 1, 1, 1, #' 1, 1, 1, 1, 1, 1, 1, 1, 1), #' tokens = c("testing", "the", "system", "second", #' "sentence", "for", "you", "system", #' "testing", "the", "tidy", "documents", #' "df", "documents", "will", "be", "parsed", #' "and", "lexranked"), #' stringsAsFactors = FALSE) #' #' df %>% #' bind_lexrank(tokens, doc_id, sent_id, level = 'tokens') #' @export bind_lexrank_ <- function(tbl, text, doc_id, sent_id=NULL, level=c("sentences", "tokens"), threshold=.2, usePageRank=TRUE, damping=0.85, continuous=FALSE, ...) { if(!is.data.frame(tbl)) stop("tbl must be a dataframe") if(!(text %in% names(tbl))) stop("text column not found in tbl") if(!(doc_id %in% names(tbl))) stop("doc_id column not found in tbl") if(!is.character(level)) stop("level must be character") if(length(level) > 1) { warning("only first element of level will be used") level = level[1] } if(!(level %in% c("sentences", "tokens"))) stop("invalid value of level; accepted values for level are 'sentences' and 'tokens'") if(level == "tokens") { if(is.null(sent_id)) stop("sent_id must be provided when level is 'tokens'") if(!(sent_id %in% names(tbl))) stop("sent_id column not found in tbl") sent_ids <- tbl[[sent_id]] } else { sent_ids <- 1:nrow(tbl) } tbl_class <- class(tbl) doc_id_class <- class(tbl[[doc_id]]) uuid_kinda <- paste0(c("a",sample(c(letters[1:6],0:9),30,replace=TRUE)), collapse = "") uuid_sep <- paste0("__", uuid_kinda,"__") doc_sent_ids <- paste0(tbl[[doc_id]], uuid_sep, sent_ids) if(level=="sentences") { sent_id <- uuid_kinda tokenDfList <- lapply(seq_along(tbl[[text]]), function(i) { sentVec <- tbl[[text]][i] tokenList <- tokenize(text = sentVec, ...) subTokenDfList <- lapply(seq_along(tokenList), function(j) { data.frame(docId=tbl[[doc_id]][i], sentenceId=doc_sent_ids[i], token=tokenList[[j]], stringsAsFactors = FALSE) }) dplyr::bind_rows(subTokenDfList) }) tokenDf <- dplyr::bind_rows(tokenDfList) %>% dplyr::filter(!is.na(token)) } else { tokenDf <- dplyr::tibble(docId=tbl[[doc_id]], sentenceId=doc_sent_ids, token=tbl[[text]]) } similDf <- sentenceSimil(tokenDf$sentenceId, tokenDf$token, tokenDf$docId) topSentIdsDf <- lexRankFromSimil(similDf$sent1, similDf$sent2, similDf$similVal, threshold=threshold, n=Inf, returnTies=TRUE, usePageRank=usePageRank, damping=damping, continuous=continuous) lex_lookup <- stringr::str_split_fixed(topSentIdsDf$sentenceId, uuid_sep, n=2) %>% dplyr::as_data_frame() %>% stats::setNames(c(doc_id, sent_id)) class(lex_lookup[[doc_id]]) <- doc_id_class lex_lookup$lexrank <- topSentIdsDf$value if(level=="tokens") { class(lex_lookup[[sent_id]]) <- class(tbl[[sent_id]]) tbl_out <- dplyr::left_join(tbl, lex_lookup, by=c(doc_id, sent_id)) } else { tbl[[uuid_kinda]] <- as.character(sent_ids) tbl_out <- dplyr::left_join(tbl, lex_lookup, by=c(doc_id, uuid_kinda)) tbl_out[[uuid_kinda]] <- NULL } class(tbl_out) <- tbl_class tbl_out } #' @rdname bind_lexrank_ #' @export bind_lexrank <- function(tbl, text, doc_id, sent_id=NULL, level=c("sentences", "tokens"), threshold=.2, usePageRank=TRUE, damping=0.85, continuous=FALSE, ...) { text_str <- as.character(substitute(text)) doc_id_str <- as.character(substitute(doc_id)) sent_id_str <- substitute(sent_id) if (!is.null(sent_id_str)) sent_id_str <- as.character(sent_id_str) bind_lexrank_(tbl, text_str, doc_id_str, sent_id=sent_id_str, level=level, threshold=threshold, usePageRank=usePageRank, damping=damping, continuous=continuous, ...) } lexRankr/R/tokenize.R0000644000175100001440000000727713056025671014254 0ustar hornikusers#' Tokenize a character vector #' Parse the elements of a character vector into a list of cleaned tokens. #' @param text The character vector to be tokenized #' @param removePunc \code{TRUE} or \code{FALSE} indicating whether or not to remove punctuation from \code{text}. If \code{TRUE}, puncuation will be removed. Defaults to \code{TRUE}. #' @param removeNum \code{TRUE} or \code{FALSE} indicating whether or not to remove numbers from \code{text}. If \code{TRUE}, numbers will be removed. Defaults to \code{TRUE}. #' @param toLower \code{TRUE} or \code{FALSE} indicating whether or not to coerce all of \code{text} to lowercase. If \code{TRUE}, \code{text} will be coerced to lowercase. Defaults to \code{TRUE}. #' @param stemWords \code{TRUE} or \code{FALSE} indicating whether or not to stem resulting tokens. If \code{TRUE}, the outputted tokens will be tokenized using \code{SnowballC::wordStem()}. Defaults to \code{TRUE}. #' @param rmStopWords \code{TRUE}, \code{FALSE}, or character vector of stopwords to remove. If \code{TRUE}, words in \code{tm::stopwords("SMART")} will be removed prior to stemming. If \code{FALSE}, no stopword removal will occur. If a character vector is passed, this vector will be used as the list of stopwords to be removed. Defaults to \code{TRUE}. #' @examples #' tokenize("Mr. Feeny said the test would be on Sat. At least I'm 99.9% sure that's what he said.") #' tokenize("Bill is trying to earn a Ph.D. in his field.", rmStopWords=FALSE) #' @importFrom magrittr "%>%" #' @export tokenize <- function(text, removePunc=TRUE, removeNum=TRUE, toLower=TRUE, stemWords=TRUE, rmStopWords=TRUE){ if(!is.character(text)) stop("text must be character") if(length(text) < 1) stop("text must be at least length 1") if(!is.logical(removePunc)) stop("removePunc must be logical") if(length(removePunc) != 1) stop("removePunc must be length 1") if(!is.logical(removeNum)) stop("removeNum must be logical") if(length(removeNum) != 1) stop("removeNum must be length 1") if(!is.logical(toLower)) stop("toLower must be logical") if(length(toLower) != 1) stop("toLower must be length 1") if(!is.logical(stemWords)) stop("stemWords must be logical") if(length(stemWords) != 1) stop("stemWords must be length 1") if(!is.logical(rmStopWords) & !is.character(rmStopWords)) stop("rmStopWords must be a logical or a character vector") if(is.character(rmStopWords)) { rmStopWordFlag <- TRUE stopwords <- rmStopWords } else if(is.logical(rmStopWords)) { if(length(rmStopWords) != 1) stop("rmStopWords must be length 1 if passed as a logical") if(rmStopWords) { rmStopWordFlag <- TRUE stopwords <- tm::stopwords("SMART") } else { rmStopWordFlag <- FALSE } } if (removePunc) text <- gsub(x=text,pattern="[^[:alnum:] ]",replacement="") if (removeNum) text <- gsub(x=text,pattern="([[:digit:]])",replacement="") if (toLower) text <- tolower(text) text <- text %>% gsub(pattern="([^[:alnum:] ])",replacement=" \\1 ") %>% gsub(pattern="\\s+",replacement=" ") %>% trimws() %>% stringr::str_split(pattern=" ") if(rmStopWordFlag) text <- lapply(text, function(tokens) { checkTokens <- tolower(tokens) if (!removePunc) { checkTokens <- gsub(x=checkTokens,pattern="[^[:alnum:] ]",replacement="") } nonStopTok <- tokens[which(!checkTokens %in% stopwords)] if(length(nonStopTok) == 0) NA_character_ else nonStopTok }) if(stemWords) { text <- lapply(text, SnowballC::wordStem) } tokenList <- lapply(text, function(tokens) { goodTok <- tokens[which(trimws(tokens) != "")] if(length(goodTok) == 0) NA_character_ else goodTok }) tokenList } lexRankr/R/lexRankFromSimil.R0000644000175100001440000001204013056025671015632 0ustar hornikusersutils::globalVariables(c("weight","sentenceId","degree","degRank")) #' Compute LexRanks from pairwise sentence similarities #' @description Compute LexRanks from sentence pair similarities using the page rank algorithm or degree centrality the methods used to compute lexRank are discussed in "LexRank: Graph-based Lexical Centrality as Salience in Text Summarization." #' @param s1 A character vector of sentence IDs corresponding to the \code{s2} and \code{simil} arguemants. #' @param s2 A character vector of sentence IDs corresponding to the \code{s1} and \code{simil} arguemants. #' @param simil A numeric vector of similiarity values that represents the similiarity between the sentences represented by the IDs in \code{s1} and \code{s2}. #' @param threshold The minimum simil value a sentence pair must have to be represented in the graph where lexRank is calculated. #' @param n The number of sentences to return as the extractive summary. The function will return the top \code{n} lexRanked sentences. See \code{returnTies} for handling ties in lexRank. #' @param returnTies \code{TRUE} or \code{FALSE} indicating whether or not to return greater than \code{n} sentence IDs if there is a tie in lexRank. If \code{TRUE}, the returned number of sentences will not be limited to \code{n}, but rather will return every sentece with a top 3 score. If \code{FALSE}, the returned number of sentences will be \code{<=n}. Defaults to \code{TRUE}. #' @param usePageRank \code{TRUE} or \code{FALSE} indicating whether or not to use the page rank algorithm for ranking sentences. If \code{FALSE}, a sentences unweighted centrality will be used as the rank. Defaults to \code{TRUE}. #' @param damping The damping factor to be passed to page rank algorithm. Ignored if \code{usePageRank} is \code{FALSE}. #' @param continuous \code{TRUE} or \code{FALSE} indicating whether or not to use continuous LexRank. Only applies if \code{usePageRank==TRUE}. If \code{TRUE}, \code{threshold} will be ignored and lexRank will be computed using a weighted graph representation of the sentences. Defaults to \code{FALSE}. #' @return A 2 column dataframe with columns \code{sentenceId} and \code{value}. \code{sentenceId} contains the ids of the top \code{n} sentences in descending order by \code{value}. \code{value} contains page rank score (if \code{usePageRank==TRUE}) or degree centrality (if \code{usePageRank==FALSE}). #' @references \url{http://www.cs.cmu.edu/afs/cs/project/jair/pub/volume22/erkan04a-html/erkan04a.html} #' @examples #' lexRankFromSimil(s1=c("d1_1","d1_1","d1_2"), s2=c("d1_2","d2_1","d2_1"), simil=c(.01,.03,.5)) #' @importFrom magrittr "%>%" #' @export lexRankFromSimil <- function(s1, s2, simil, threshold=.2, n=3, returnTies=TRUE, usePageRank=TRUE, damping=0.85, continuous=FALSE) { if(!is.logical(returnTies)) stop("returnTies must be logical") if(length(returnTies) != 1) stop("returnTies must be length 1") if(!is.logical(usePageRank)) stop("usePageRank must be logical") if(length(usePageRank) != 1) stop("usePageRank must be length 1") if(!is.logical(continuous)) stop("continuous must be logical") if(length(continuous) != 1) stop("continuous must be length 1") if(!is.numeric(simil)) stop("simil must be numeric") if(!is.numeric(n)) stop("n must be numeric") if(length(n) != 1) stop("n must be length 1") if (length(s1) != length(s2) | length(s1) != length(simil)) stop("s1, s2, & simil must all be the same length") if (sum(simil) == 0) stop("all simil values are zero") if (sum(simil > threshold) == 0) stop("all simil values are below threshold") s1 <- as.character(s1) s2 <- as.character(s2) if(returnTies) tieMethod <- "min" else if(!returnTies) tieMethod <- "first" edges <- dplyr::data_frame(s1=s1, s2=s2, weight=simil) if(!continuous | !usePageRank) { if(!is.numeric(threshold)) stop("threshold must be numeric") if(length(threshold) != 1) stop("threshold must be length 1") edges <- edges %>% dplyr::filter(weight > threshold) %>% dplyr::select(-weight) } if (usePageRank) { if(!is.numeric(damping)) stop("damping must be numeric") if(length(damping) != 1) stop("damping must be length 1") sentGraph <- igraph::graph_from_data_frame(edges, directed = FALSE) sentRank <- igraph::page_rank(sentGraph, directed=FALSE)$vector sentRanksRanked <- rank(1/sentRank, ties.method = tieMethod) topCentral <- sentRank[which(sentRanksRanked <= n)] centralDf <- data.frame(sentenceId=names(topCentral), value=topCentral,stringsAsFactors = FALSE) rownames(centralDf) <- NULL } else if(!usePageRank){ centralDf <- dplyr::data_frame(sentenceId = c(edges$s1, edges$s2)) %>% dplyr::group_by(sentenceId) %>% dplyr::summarise(degree=n()) %>% dplyr::ungroup() %>% dplyr::arrange(dplyr::desc(degree)) %>% dplyr::mutate(degRank = rank(1/degree, ties.method = tieMethod)) %>% dplyr::filter(degRank <= n) %>% dplyr::select(sentenceId, value=degree) class(centralDf) <- "data.frame" } return(centralDf) } lexRankr/R/RcppExports.R0000644000175100001440000000035313056025671014701 0ustar hornikusers# Generated by using Rcpp::compileAttributes() -> do not edit by hand # Generator token: 10BE3573-1514-4C36-9D1C-5A225CD40393 idfCosineSimil <- function(mat) { .Call('lexRankr_idfCosineSimil', PACKAGE = 'lexRankr', mat) } lexRankr/R/lexRank.R0000644000175100001440000001320113056025671014010 0ustar hornikusersutils::globalVariables(c("value")) #' Extractive text summarization with LexRank #' @description Compute LexRanks from a vector of documents using the page rank algorithm or degree centrality the methods used to compute lexRank are discussed in "LexRank: Graph-based Lexical Centrality as Salience in Text Summarization." #' @param text A character vector of documents to be cleaned and processed by the LexRank algorithm #' @param docId A vector of document IDs with length equal to the length of \code{text}. If \code{docId == "create"} then doc IDs will be created as an index from 1 to \code{n}, where \code{n} is the length of \code{text}. #' @param threshold The minimum simil value a sentence pair must have to be represented in the graph where lexRank is calculated. #' @param n The number of sentences to return as the extractive summary. The function will return the top \code{n} lexRanked sentences. See \code{returnTies} for handling ties in lexRank. #' @param returnTies \code{TRUE} or \code{FALSE} indicating whether or not to return greater than \code{n} sentence IDs if there is a tie in lexRank. If \code{TRUE}, the returned number of sentences will not be limited to \code{n}, but rather will return every sentece with a top 3 score. If \code{FALSE}, the returned number of sentences will be \code{<=n}. Defaults to \code{TRUE}. #' @param usePageRank \code{TRUE} or \code{FALSE} indicating whether or not to use the page rank algorithm for ranking sentences. If \code{FALSE}, a sentences unweighted centrality will be used as the rank. Defaults to \code{TRUE}. #' @param damping The damping factor to be passed to page rank algorithm. Ignored if \code{usePageRank} is \code{FALSE}. #' @param continuous \code{TRUE} or \code{FALSE} indicating whether or not to use continuous LexRank. Only applies if \code{usePageRank==TRUE}. If \code{TRUE}, \code{threshold} will be ignored and lexRank will be computed using a weighted graph representation of the sentences. Defaults to \code{FALSE}. #' @param sentencesAsDocs \code{TRUE} or \code{FALSE}, indicating whether or not to treat sentences as documents when calculating tfidf scores for similarity. If \code{TRUE}, inverse document frequency will be calculated as inverse sentence frequency (useful for single document extractive summarization). #' @param removePunc \code{TRUE} or \code{FALSE} indicating whether or not to remove punctuation from text while tokenizing. If \code{TRUE}, puncuation will be removed. Defaults to \code{TRUE}. #' @param removeNum \code{TRUE} or \code{FALSE} indicating whether or not to remove numbers from text while tokenizing. If \code{TRUE}, numbers will be removed. Defaults to \code{TRUE}. #' @param toLower \code{TRUE} or \code{FALSE} indicating whether or not to coerce all of text to lowercase while tokenizing. If \code{TRUE}, \code{text} will be coerced to lowercase. Defaults to \code{TRUE}. #' @param stemWords \code{TRUE} or \code{FALSE} indicating whether or not to stem resulting tokens. If \code{TRUE}, the outputted tokens will be tokenized using \code{SnowballC::wordStem()}. Defaults to \code{TRUE}. #' @param rmStopWords \code{TRUE}, \code{FALSE}, or character vector of stopwords to remove from tokens. If \code{TRUE}, words in \code{tm::stopwords("SMART")} will be removed prior to stemming. If \code{FALSE}, no stopword removal will occur. If a character vector is passed, this vector will be used as the list of stopwords to be removed. Defaults to \code{TRUE}. #' @param Verbose \code{TRUE} or \code{FALSE} indicating whether or not to \code{cat} progress messages to the console while running. Defaults to \code{TRUE}. #' @return A 2 column dataframe with columns \code{sentenceId} and \code{value}. \code{sentence} contains the ids of the top \code{n} sentences in descending order by \code{value}. \code{value} contains page rank score (if \code{usePageRank==TRUE}) or degree centrality (if \code{usePageRank==FALSE}). #' @references \url{http://www.cs.cmu.edu/afs/cs/project/jair/pub/volume22/erkan04a-html/erkan04a.html} #' @examples #' lexRank(c("This is a test.","Tests are fun.", #' "Do you think the exam will be hard?","Is an exam the same as a test?", #' "How many questions are going to be on the exam?")) #' @importFrom magrittr "%>%" #' @export lexRank <- function(text, docId = "create", threshold=.2, n=3, returnTies=TRUE, usePageRank=TRUE, damping=0.85, continuous=FALSE, sentencesAsDocs=FALSE, removePunc=TRUE, removeNum=TRUE, toLower=TRUE, stemWords=TRUE, rmStopWords=TRUE, Verbose=TRUE){ if(!is.logical(Verbose)) stop("Verbose must be logical") if(length(Verbose) != 1) stop("Verbose must be length 1") if(Verbose) cat("Parsing text into sentences and tokens...") sentTokList <- sentenceTokenParse(text=text, docId = docId, removePunc=removePunc, removeNum=removeNum, toLower=toLower, stemWords=stemWords, rmStopWords=rmStopWords) if(Verbose) cat("DONE\n") sentDf <- sentTokList$sentences tokenDf <- sentTokList$tokens if(Verbose) cat("Calculating pairwise sentence similarities...") similDf <- sentenceSimil(sentenceId=tokenDf$sentenceId, token=tokenDf$token, docId=tokenDf$docId, sentencesAsDocs=sentencesAsDocs) if(Verbose) cat("DONE\n") if(Verbose) cat("Applying LexRank...") topNSents <- lexRankFromSimil(s1=similDf$sent1, s2=similDf$sent2, simil=similDf$similVal, threshold=threshold, n=n, returnTies=returnTies, usePageRank=usePageRank, damping=damping, continuous=continuous) if(Verbose) cat("DONE\nFormatting Output...") returnDf <- sentDf %>% dplyr::inner_join(topNSents, by=c("sentenceId"="sentenceId")) %>% dplyr::arrange(dplyr::desc(value)) if(Verbose) cat("DONE\n") return(returnDf) } lexRankr/R/unnest_sentences.R0000644000175100001440000000406113056025671015773 0ustar hornikusers#' Split a column of text into sentences #' @description Split a column of text into sentences #' @param tbl dataframe containing column of text to be split into sentences #' @param output name of column to be created to store parsed sentences #' @param input name of input column of text to be parsed into sentences #' @param output_id name of column to be created to store sentence ids #' @param drop whether original input column should get dropped #' @return A data.frame of parsed sentences and sentence ids #' @examples #' library(dplyr) #' #' df <- dplyr::tibble(doc_id = 1:3, #' text = c("Testing the system. Second sentence for you.", #' "System testing the tidy documents df.", #' "Documents will be parsed and lexranked.")) #' #' unnest_sentences(df, sents, text) #' unnest_sentences_(df, "sents", "text") #' #' df %>% #' unnest_sentences(sents, text) #' @export unnest_sentences_ <- function(tbl, output, input, output_id="sent_id", drop=TRUE) { if(!is.data.frame(tbl)) stop("tbl must be a dataframe") if(!(input %in% names(tbl))) stop("input column not found in tbl") if(!is.character(tbl[[input]])) stop("input column must be character") if(length(output_id) > 1) { warning("only first element of output_id will be used") output_id <- output_id[1] } if(!is.logical(drop)) stop("drop must be logical") text <- tbl[[input]] parsed_sents <- sentence_parser(text) sent_ids <- lapply(parsed_sents, function(.x) 1:length(.x)) if (drop) { tbl[[input]] <- NULL } tbl[[output_id]] <- sent_ids tbl[[output]] <- parsed_sents tidyr::unnest(tbl) } #' @rdname unnest_sentences_ #' @export unnest_sentences <- function(tbl, output, input, output_id='sent_id', drop=TRUE) { output_str <- as.character(substitute(output)) input_str <- as.character(substitute(input)) out_id_str <- as.character(substitute(output_id)) unnest_sentences_(tbl, output_str, input_str, out_id_str, drop) } lexRankr/R/sentenceSimil.R0000644000175100001440000001123113056025671015207 0ustar hornikusers#' @useDynLib lexRankr #' @importFrom Rcpp sourceCpp NULL utils::globalVariables(c("n","tf","idf","tfidf","V1","V2","similVal")) #' Compute distance between sentences #' @description Compute distance between sentences using modified idf cosine distance from "LexRank: Graph-based Lexical Centrality as Salience in Text Summarization". Output can be used as input to \code{\link{lexRankFromSimil}}. #' @param sentenceId A character vector of sentence IDs corresponding to the \code{docId} and \code{token} arguemants. #' @param token A character vector of tokens corresponding to the \code{docId} and \code{sentenceId} arguemants. #' @param docId A character vector of document IDs corresponding to the \code{sentenceId} and \code{token} arguemants. Can be \code{NULL} if \code{sentencesAsDocs} is \code{TRUE}. #' @param sentencesAsDocs \code{TRUE} or \code{FALSE}, indicating whether or not to treat sentences as documents when calculating tfidf scores. If \code{TRUE}, inverse document frequency will be calculated as inverse sentence frequency (useful for single document extractive summarization) #' @return A 3 column dataframe of pairwise distances between sentences. Columns: \code{sent1} (sentence id), \code{sent2} (sentence id), & \code{dist} (distance between \code{sent1} and \code{sent2}). #' @references \url{http://www.cs.cmu.edu/afs/cs/project/jair/pub/volume22/erkan04a-html/erkan04a.html} #' @examples #' sentenceSimil(docId=c("d1","d1","d2","d2"), #' sentenceId=c("d1_1","d1_1","d2_1","d2_1"), #' token=c("i", "ran", "jane", "ran")) #' @importFrom utils combn #' @importFrom magrittr "%>%" #' @export sentenceSimil <- function(sentenceId, token, docId=NULL, sentencesAsDocs=FALSE){ if(!is.logical(sentencesAsDocs)) stop("sentencesAsDocs must be logical") if(length(sentencesAsDocs) != 1) stop("sentencesAsDocs must be length 1") if(!sentencesAsDocs & is.null(docId)) stop("docIds must be provided if sentencesAsDocs is FALSE") sentenceId <- as.character(sentenceId) if(!is.character(token)) stop("token must be character") if(length(token) < 1) stop("token must be at least length 1") if(sentencesAsDocs) { docId <- sentenceId if(length(docId) != length(sentenceId) | length(docId) != length(token)) stop("docId, sentenceId, & token must all be the same length") } else if (!sentencesAsDocs) { docId <- as.character(docId) if(length(sentenceId) != length(token)) stop("sentenceId & token must be the same length") } ndoc <- length(unique(docId)) if(ndoc > length(unique(sentenceId))) warning("There are more unique docIds than sentenceIds. Verify you have passed the correct parameters to the function.") tokenDf <- dplyr::data_frame(docId=docId, sentenceId=sentenceId, token=token) stm <- tokenDf %>% dplyr::group_by(docId,token) %>% dplyr::summarise(tf = n()) %>% dplyr::ungroup() %>% dplyr::group_by(token) %>% dplyr::mutate(idf = log(ndoc/n())) %>% dplyr::mutate(tfidf = tf*idf) %>% dplyr::ungroup() if(!sentencesAsDocs) { stm <- stm %>% dplyr::right_join(tokenDf, by=c("docId"="docId", "token"="token")) %>% dplyr::select(sentenceId, token, tfidf) %>% dplyr::filter(tfidf > 0) %>% unique() } else if (sentencesAsDocs) { stm <- stm %>% dplyr::select(sentenceId=docId, token, tfidf) %>% dplyr::filter(tfidf > 0) %>% unique() } if(nrow(stm)==0) stop("All values in sentence term tfidf matrix are 0. Similarities would return as NaN") if(length(unique((stm$sentenceId))) == 1) stop("Only one sentence had nonzero tfidf scores. Similarities would return as NaN") stm <- tidyr::spread(stm, key=token, value=tfidf, fill=0, drop=FALSE) matRowNames <- stm$sentenceId stm <- stm %>% dplyr::select(-sentenceId) %>% as.matrix() rownames(stm) <- matRowNames #old non C slow version for idfcosine similarity # idfCosine <- function(x,y) { # sum(x*y)/(sqrt(sum(x^2))*sqrt(sum(y^2))) # } # prDBname <- "idfCosine" # while (proxy::pr_DB$entry_exists(prDBname)) { # prDBname <- paste0(prDBname, sample(100:999,1)) # cat(prDBname,"\n") # } # proxy::pr_DB$set_entry(FUN=idfCosine, names=prDBname) # similMat <- proxy::dist(stm, method=prDBname) # proxy::pr_DB$delete_entry(prDBname) sentencePairsDf <- sort(rownames(stm)) %>% combn(2) %>% t() %>% as.data.frame(stringsAsFactors=FALSE) %>% dplyr::mutate(similVal = idfCosineSimil(stm)) %>% # dplyr::mutate(similVal = as.numeric(similMat)) %>% dplyr::select(sent1=V1, sent2=V2, similVal) class(sentencePairsDf) <- "data.frame" return(sentencePairsDf) } lexRankr/R/sentenceParse.R0000644000175100001440000000375413056025671015217 0ustar hornikusers#' Parse text into sentences #' @description Parse the elements of a character vector into a dataframe of sentences with additional identifiers. #' @param text Character vector to be parsed into sentences #' @param docId A vector of document IDs with length equal to the length of \code{text}. If \code{docId == "create"} then doc IDs will be created as an index from 1 to \code{n}, where \code{n} is the length of \code{text}. #' @return A data frame with 3 columns and \code{n} rows, where \code{n} is the number of sentences found by the routine. Column 1: \code{docId} document id for the sentence. Column 2: \code{sentenceId} sentence id for the sentence. Column 3: \code{sentence} the sentences found in the routine. #' @examples #' sentenceParse("Bill is trying to earn a Ph.D.", "You have to have a 5.0 GPA.") #' sentenceParse(c("Bill is trying to earn a Ph.D.", "You have to have a 5.0 GPA."), #' docId=c("d1","d2")) #' @importFrom magrittr "%>%" #' @export sentenceParse <- function(text, docId = "create") { if(!is.character(text)) stop("text must be character") if(length(text) < 1) stop("text must be at least length 1") docId <- as.character(docId) if(length(docId)==1 & docId[1]=="create") { createDocIds <- TRUE } else if(length(docId)==length(text)) { createDocIds <- FALSE } else if(length(docId)!=length(text)) stop("docId vector must be same length as text vector") sentences <- sentence_parser(text) sentenceDfList <- lapply(seq_along(sentences), function(i) { sentVec <- trimws(sentences[[i]]) if(createDocIds) { data.frame(docId=i, sentenceId=paste0(i,"_",seq_along(sentVec)), sentence=sentVec, stringsAsFactors = FALSE) } else if(!createDocIds) { data.frame(docId=docId[i], sentenceId=paste0(docId[i],"_",seq_along(sentVec)), sentence=sentVec, stringsAsFactors = FALSE) } }) sentenceDf <- dplyr::bind_rows(sentenceDfList) class(sentenceDf) <- "data.frame" return(sentenceDf) } lexRankr/R/sentence_parser.R0000644000175100001440000000101513056025671015564 0ustar hornikusers#' Utility to parse sentences from text #' @description Utility to parse sentences from text; created to have a central shared sentence parsing function #' @param text Character vector to be parsed into sentences #' @return A list with length equal to `length(text)`; list elements are character vectors of text parsed with sentence regex sentence_parser <- function(text) { stringr::str_split(string = text, pattern = stringr::regex("(?% unnest_sentences(sents, text) %>% bind_lexrank(sents, doc_id, level = 'sentences') %>% arrange(desc(lexrank)) ``` ####lexRank applied to a charcter vector of documents ``` library(lexRankr) text <- c("Testing the system. Second sentence for you.", "System testing the tidy documents df.", "Documents will be parsed and lexranked.") lexRank(text) ``` ##Example with Twitter (using lexRank helper functions) ``` library(jsonlite) library(httr) library(lexRankr) ######################################################### #FUNCTION TO GET TEXT OF USERS LAST N TWEETS ON TIMELINE# ######################################################### getUserTweets <- function(user, n, consKey, consSecret, token, tokenSecret) { #SET UP FOR API auth <- oauth_app("twitter", key=consKey, secret=consSecret) sig <- sign_oauth1.0(auth, token=token, token_secret=tokenSecret) #INITIALIZE COUNTERS AND STORAGE nLeft <- n i <- 0 tweetText <- character(0) #LOOP UNTIL n IS MET while (nLeft > 0) { nToGet <- min(200, nLeft) i <- i+1 #SET MAX ID IF i > 1 (MAX ID WILL KEEP TWEETS FROM BEING DUPLICATED IN GET) if (i==1) { GETurl <- paste0("https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=", user,"&count=", nToGet) } else { GETurl <- paste0("https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=", user,"&count=",nToGet,"&max_id=", max_id) } #GET TWEETS timelineRaw <- GET(GETurl, sig) #EXTRACT TWEET TEXT FROM GET timelineContent <- content(timelineRaw) jsonRaw <- toJSON(timelineContent) jsonConv <- iconv(jsonRaw, "UTF-8", "ASCII", sub = "") #clean out emoji and other special chars jsonConv <- gsub("\003", "", jsonConv) #special character (^C) not caught by above clean jsonClean <- fromJSON(jsonConv) #APPEND TWEET TEXT tweetText <- c(tweetText, unlist(jsonClean$text)) #STORE MAX ID FOR USE IN NEXT GETurl suppressWarnings(max_id <- min(unlist(jsonClean$id))) #UPDATE NUMBER OF TWEETS LEFT TO MEET n PARAMETER nLeft <- nLeft-nToGet } return(unique(tweetText)) } ################################################################################## #FUNCTION USING LEXRANKR TO FIND THE MOST REPRESENTATIVE TWEETS OF USERS TIMELINE# ################################################################################## tweetRankr <- function(tweetText, dropMentions=TRUE, dropHashtags=TRUE, n=5, returnTies=TRUE, printTweets = TRUE) { #store original tweet text tweetTextOg <- tweetText #remove instances of @username from text if (dropMentions) tweetText <- gsub("\\@\\w+","",tweetText) #remove insances of #topic from text if (dropHashtags) tweetText <- gsub("\\#\\w+","",tweetText) #clean up any multiple spaces introduced by modifying text tweetText <- trimws(gsub("\\s+", " ", tweetText)) #parse text and create doc/sent Ids for each token sentenceTokenList <- lexRankr::sentenceTokenParse(tweetText) #store token data.frame from list output #token data.frame has columns for the token, document Id, and sentence Id tweetTokenDf <- sentenceTokenList$tokens #compute pairwise tweet similiarity #using document id from token df instead of sentence id #using docId will find most central tweets as opposed to most central sentences within tweets similDf <- lexRankr::sentenceSimil(tweetTokenDf$docId, tweetTokenDf$token, tweetTokenDf$docId) #apply lexRank algorithm to return top n tweet ids topTweetIdsDf <- lexRankr::lexRankFromSimil(similDf$sent1, similDf$sent2, similDf$similVal, n = n, returnTies = returnTies) #subset tweet text variable to extract most central tweets according to lexrank topTweets <- tweetTextOg[as.numeric(topTweetIdsDf$sentenceId)] if (printTweets) cat(paste(paste0(1:length(topTweets), ") ",topTweets), collapse="\n\n")) invisible(return(topTweets)) } consKey <- 'consumer key' consSecret <- 'consumer secret' token <- 'access token' tokenSecret <- 'access token secret' ################### # Hillary Clinton # ################### tweetTextHill <- getUserTweets(user="HillaryClinton", n=5000, consKey, consSecret, token, tokenSecret) topTweetsHill <- tweetRankr(tweetTextHill) ``` 1) "Hillary Clinton must become the next president of the United States. @BernieSanders #DemsInPhilly 2) "In this election, Im with her!" @FLOTUS on Hillary #DemsInPhilly 3) We are better than this. 4) "As your president, I will always have your back." Hillary 5) She knows that love trumps hate. @POTUS on Hillary ``` ################ # Donald Trump # ################ tweetTextTrump <- getUserTweets(user="realDonaldTrump", n=5000, consKey, consSecret, token, tokenSecret) topTweetsTrump <- tweetRankr(tweetTextTrump) ``` 1) "@tcloer11: @realDonaldTrump Great job! Make America Great Again!" 2) Wisconsin, we will MAKE AMERICA GREAT AGAIN! 3) MAKE AMERICA GREAT AGAIN! 4) MAKE AMERICA GREAT AGAIN! MAKE AMERICA SAFE AGAIN! 5) AMERICA FIRST! ``` ################## # Bernie Sanders # ################## tweetTextBern <- getUserTweets(user="SenSanders", n=5000, consKey, consSecret, token, tokenSecret) topTweetsBern <- tweetRankr(tweetTextBern) ``` 1) Unless Congress stands up for the middle class that's getting stepped on by the billionaire class, soon there won't be aiddle class left. 2) The current federal minimum wage of $7.25 an hour is a starvation wage and must be raised. The minimum wage must become aiving wage. 3) Weve got to stand up to the fossil fuel industry and fight for legislation that transforms our energy system away from fossiluels. 4) There is no justice when so few have so much and so many have so little. 5) Health care is a right, not a privilege. Everyone in America should be able to access the health care they need regardless of their income. ``` ############# # Rbloggers # ############# tweetTextRblog <- getUserTweets(user="Rbloggers", n=5000, consKey, consSecret, token, tokenSecret) topTweetsRblog <- tweetRankr(tweetTextRblog) ``` 1) New R job: Data Scientist Machine Learning https://t.co/YiWwXkmxmc #rstats #DataScience #jobs 2) New R job: Principal Analysts x2, Senior Analyst, Analyst (@ Wellington ) http://t.co/5OLIDl51tw #rstats #jobs 3) A Few Days of Python: Using R in Python http://t.co/28j8CAYThn #rstats 4) Network visualization part 4: 3D networks https://t.co/U6U53xG679 #rstats #DataScience 5) Network visualization part 4: 3D networks https://t.co/Y625xNNr03 #rstats #DataScience lexRankr/MD50000644000175100001440000000474013056034162012373 0ustar hornikusersb09fa890f724307b7a666da0ae18b966 *DESCRIPTION b458a8dd25dc372c4aeefb1b28542c7c *LICENSE 3dfbe6abf64ff40c99c68b366de7ecac *NAMESPACE 7ddd5f4b328baf74b02ff7fbf3360715 *NEWS.md 8e9822c41154fdc3672feb037b3de49a *R/RcppExports.R 39017020f0c1b46e32112b5a6ef3bf5a *R/bind_lexrank.R b5b5b38a51a0744b75921752089613ca *R/lexRank.R a32b76ade6f7d0c12ef34bb16fb357e9 *R/lexRankFromSimil.R 9b512a4022f5cb870154ec8e59271f53 *R/sentenceParse.R 308eaca464fb90969f0e02c4ec70070d *R/sentenceSimil.R 6c276fb56c0ee80e3f74e59eb6e49c41 *R/sentenceTokenParse.R c6d46d77b1da31caeff8e47bc8ec7a1d *R/sentence_parser.R 73ae0bc326c31900e2dde09b20af5aba *R/tokenize.R 06a7631f18bac7fbf85675ad22d5abba *R/unnest_sentences.R af5d573f1dc77a5aab14b8cd6e8747aa *README.md e7c8e205a5d20da21a8a2656f9ef1d32 *build/vignette.rds 85e967ef3b2094b02fd3fac0d0b1968a *inst/doc/Analyzing_Twitter_with_LexRankr.html 66e47bc11bb84e17ca6e0e3c79e18d73 *inst/doc/Analyzing_Twitter_with_LexRankr.html.asis 861cae03e1e1404451d636908c50fd4a *man/bind_lexrank_.Rd 61567a82d7fb41bf03d0c46ac87da8d0 *man/lexRank.Rd 3409c9b9026182386a6123fc90aa39e8 *man/lexRankFromSimil.Rd 59d7a80fbb8dd83bdcd3b6e159db33f6 *man/sentenceParse.Rd 55d9073db96b6c6ecca25de323b40989 *man/sentenceSimil.Rd 9bbcd1ae27f2c5d8c1fcb6f0c8a72cfb *man/sentenceTokenParse.Rd 60c25a4a9d91c9c605bb6a1daaa13e3e *man/sentence_parser.Rd 2b28bf4eb65e4369816f5c42d86b9a73 *man/tokenize.Rd 52c68d97fe2308c39781617d0c1264e5 *man/unnest_sentences_.Rd a4ba729353921f27dd1033666ba3d5a7 *src/RcppExports.cpp 2cd12945f1ba427de73a8b690ffd4615 *src/idfCosineSimil.cpp b911f4ffbf1ed05047d75ba046bb4b54 *src/register_routines.c f2f13352ba2520410e0befed450da512 *tests/testthat.R fdb5c9a4e1b4fdc945f60e13e357d2fa *tests/testthat/test-bind_lexrank.R 8cbdc84080d998fb116e1879865551cf *tests/testthat/test-bind_lexrank_.R af81d75390b209670947a0e5aa241719 *tests/testthat/test-idfCosine.R 7e63f7f2a086f6063c363f4c566e25e4 *tests/testthat/test-lexRank.R a2a11099148d09929fb88c23d0291cbb *tests/testthat/test-lexRankFromSimil.R 47c695bc1da291f010842fbcdbb1cb4a *tests/testthat/test-sentenceParse.R eae78dcfea05975cb1f37ab8659f91f1 *tests/testthat/test-sentenceSimil.R 95d266d769c968dbc8680bf18925d02d *tests/testthat/test-sentenceTokenParse.R 9aeddf2207014720bc16cf27c5c5b3a0 *tests/testthat/test-tokenize.R b8cb8bf2c1ab67737de55e66fe433cac *tests/testthat/test-unnest_sentences.R 402d44fcccf8e09d18cbe4a5a6656810 *tests/testthat/test-unnest_sentences_.R 66e47bc11bb84e17ca6e0e3c79e18d73 *vignettes/Analyzing_Twitter_with_LexRankr.html.asis lexRankr/build/0000755000175100001440000000000013056025704013157 5ustar hornikuserslexRankr/build/vignette.rds0000644000175100001440000000036013056025704015515 0ustar hornikusersP 0N ť DJh14PuՋmDŃ}n%P%A`}Ѐ~<\]RU)&sq^r}0qf*,pF50rQҏGyqzwV 괭T&gRƏi=T*i}'lL0Xb?똼=Qwn~z lexRankr/DESCRIPTION0000644000175100001440000000132013056034162013560 0ustar hornikusersPackage: lexRankr Type: Package Title: Extractive Summarization of Text with the LexRank Algorithm Version: 0.4.0 Author: Adam Spannbauer Maintainer: Adam Spannbauer Description: An R implementation of the LexRank algorithm described by G. Erkan and D. R. Radev (2004) . License: MIT + file LICENSE URL: https://github.com/AdamSpannbauer/lexRankr/ LazyData: TRUE RoxygenNote: 5.0.1 Imports: dplyr, tidyr, magrittr, stringr, SnowballC, igraph, tm, Rcpp LinkingTo: Rcpp Suggests: covr, testthat, R.rsp VignetteBuilder: R.rsp NeedsCompilation: yes Packaged: 2017-03-02 14:27:17 UTC; u775749 Repository: CRAN Date/Publication: 2017-03-02 16:21:22 lexRankr/man/0000755000175100001440000000000013056025671012636 5ustar hornikuserslexRankr/man/bind_lexrank_.Rd0000644000175100001440000000676313056025671015740 0ustar hornikusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/bind_lexrank.R \name{bind_lexrank_} \alias{bind_lexrank} \alias{bind_lexrank_} \title{Bind lexrank scores to a dataframe of text} \usage{ bind_lexrank_(tbl, text, doc_id, sent_id = NULL, level = c("sentences", "tokens"), threshold = 0.2, usePageRank = TRUE, damping = 0.85, continuous = FALSE, ...) bind_lexrank(tbl, text, doc_id, sent_id = NULL, level = c("sentences", "tokens"), threshold = 0.2, usePageRank = TRUE, damping = 0.85, continuous = FALSE, ...) } \arguments{ \item{tbl}{dataframe containing column of sentences to be lexranked} \item{text}{name of column containing sentences or tokens to be lexranked} \item{doc_id}{name of column containing document ids corresponding to \code{text}} \item{sent_id}{Only needed if \code{level} is "tokens". name of column containing sentence ids corresponding to \code{text}} \item{level}{the parsed level of the text column to be lexranked. i.e. is \code{text} a column of "sentences" or "tokens"? The "tokens" level is provided to allow users to implement custom tokenization. Note: even if the input \code{level} is "tokens" lexrank scores are assigned at the sentence level.} \item{threshold}{The minimum simililarity value a sentence pair must have to be represented in the graph where lexRank is calculated.} \item{usePageRank}{\code{TRUE} or \code{FALSE} indicating whether or not to use the page rank algorithm for ranking sentences. If \code{FALSE}, a sentences unweighted centrality will be used as the rank. Defaults to \code{TRUE}.} \item{damping}{The damping factor to be passed to page rank algorithm. Ignored if \code{usePageRank} is \code{FALSE}.} \item{continuous}{\code{TRUE} or \code{FALSE} indicating whether or not to use continuous LexRank. Only applies if \code{usePageRank==TRUE}. If \code{TRUE}, \code{threshold} will be ignored and lexRank will be computed using a weighted graph representation of the sentences. Defaults to \code{FALSE}.} \item{...}{tokenizing options to be passed to lexRankr::tokenize. Ignored if \code{level} is "sentences"} } \value{ A dataframe with an additional column of lexrank scores (column is given name lexrank) } \description{ Bind lexrank scores to a dataframe of sentences or to a dataframe of tokens with sentence ids } \examples{ library(dplyr) df <- dplyr::tibble(doc_id = 1:3, text = c("Testing the system. Second sentence for you.", "System testing the tidy documents df.", "Documents will be parsed and lexranked.")) df \%>\% unnest_sentences(sents, text) \%>\% bind_lexrank(sents, doc_id, level = 'sentences') df \%>\% unnest_sentences(sents, text) \%>\% bind_lexrank_("sents", "doc_id", level = 'sentences') df <- data.frame(doc_id = c(1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3), sent_id = c(1, 1, 1, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1), tokens = c("testing", "the", "system", "second", "sentence", "for", "you", "system", "testing", "the", "tidy", "documents", "df", "documents", "will", "be", "parsed", "and", "lexranked"), stringsAsFactors = FALSE) df \%>\% bind_lexrank(tokens, doc_id, sent_id, level = 'tokens') } lexRankr/man/sentence_parser.Rd0000644000175100001440000000104513056025671016305 0ustar hornikusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/sentence_parser.R \name{sentence_parser} \alias{sentence_parser} \title{Utility to parse sentences from text} \usage{ sentence_parser(text) } \arguments{ \item{text}{Character vector to be parsed into sentences} } \value{ A list with length equal to `length(text)`; list elements are character vectors of text parsed with sentence regex } \description{ Utility to parse sentences from text; created to have a central shared sentence parsing function } lexRankr/man/sentenceParse.Rd0000644000175100001440000000221113056025671015720 0ustar hornikusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/sentenceParse.R \name{sentenceParse} \alias{sentenceParse} \title{Parse text into sentences} \usage{ sentenceParse(text, docId = "create") } \arguments{ \item{text}{Character vector to be parsed into sentences} \item{docId}{A vector of document IDs with length equal to the length of \code{text}. If \code{docId == "create"} then doc IDs will be created as an index from 1 to \code{n}, where \code{n} is the length of \code{text}.} } \value{ A data frame with 3 columns and \code{n} rows, where \code{n} is the number of sentences found by the routine. Column 1: \code{docId} document id for the sentence. Column 2: \code{sentenceId} sentence id for the sentence. Column 3: \code{sentence} the sentences found in the routine. } \description{ Parse the elements of a character vector into a dataframe of sentences with additional identifiers. } \examples{ sentenceParse("Bill is trying to earn a Ph.D.", "You have to have a 5.0 GPA.") sentenceParse(c("Bill is trying to earn a Ph.D.", "You have to have a 5.0 GPA."), docId=c("d1","d2")) } lexRankr/man/sentenceSimil.Rd0000644000175100001440000000325613056025671015735 0ustar hornikusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/sentenceSimil.R \name{sentenceSimil} \alias{sentenceSimil} \title{Compute distance between sentences} \usage{ sentenceSimil(sentenceId, token, docId = NULL, sentencesAsDocs = FALSE) } \arguments{ \item{sentenceId}{A character vector of sentence IDs corresponding to the \code{docId} and \code{token} arguemants.} \item{token}{A character vector of tokens corresponding to the \code{docId} and \code{sentenceId} arguemants.} \item{docId}{A character vector of document IDs corresponding to the \code{sentenceId} and \code{token} arguemants. Can be \code{NULL} if \code{sentencesAsDocs} is \code{TRUE}.} \item{sentencesAsDocs}{\code{TRUE} or \code{FALSE}, indicating whether or not to treat sentences as documents when calculating tfidf scores. If \code{TRUE}, inverse document frequency will be calculated as inverse sentence frequency (useful for single document extractive summarization)} } \value{ A 3 column dataframe of pairwise distances between sentences. Columns: \code{sent1} (sentence id), \code{sent2} (sentence id), & \code{dist} (distance between \code{sent1} and \code{sent2}). } \description{ Compute distance between sentences using modified idf cosine distance from "LexRank: Graph-based Lexical Centrality as Salience in Text Summarization". Output can be used as input to \code{\link{lexRankFromSimil}}. } \examples{ sentenceSimil(docId=c("d1","d1","d2","d2"), sentenceId=c("d1_1","d1_1","d2_1","d2_1"), token=c("i", "ran", "jane", "ran")) } \references{ \url{http://www.cs.cmu.edu/afs/cs/project/jair/pub/volume22/erkan04a-html/erkan04a.html} } lexRankr/man/unnest_sentences_.Rd0000644000175100001440000000242713056025671016654 0ustar hornikusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/unnest_sentences.R \name{unnest_sentences_} \alias{unnest_sentences} \alias{unnest_sentences_} \title{Split a column of text into sentences} \usage{ unnest_sentences_(tbl, output, input, output_id = "sent_id", drop = TRUE) unnest_sentences(tbl, output, input, output_id = "sent_id", drop = TRUE) } \arguments{ \item{tbl}{dataframe containing column of text to be split into sentences} \item{output}{name of column to be created to store parsed sentences} \item{input}{name of input column of text to be parsed into sentences} \item{output_id}{name of column to be created to store sentence ids} \item{drop}{whether original input column should get dropped} } \value{ A data.frame of parsed sentences and sentence ids } \description{ Split a column of text into sentences } \examples{ library(dplyr) df <- dplyr::tibble(doc_id = 1:3, text = c("Testing the system. Second sentence for you.", "System testing the tidy documents df.", "Documents will be parsed and lexranked.")) unnest_sentences(df, sents, text) unnest_sentences_(df, "sents", "text") df \%>\% unnest_sentences(sents, text) } lexRankr/man/lexRankFromSimil.Rd0000644000175100001440000000552413056025671016361 0ustar hornikusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/lexRankFromSimil.R \name{lexRankFromSimil} \alias{lexRankFromSimil} \title{Compute LexRanks from pairwise sentence similarities} \usage{ lexRankFromSimil(s1, s2, simil, threshold = 0.2, n = 3, returnTies = TRUE, usePageRank = TRUE, damping = 0.85, continuous = FALSE) } \arguments{ \item{s1}{A character vector of sentence IDs corresponding to the \code{s2} and \code{simil} arguemants.} \item{s2}{A character vector of sentence IDs corresponding to the \code{s1} and \code{simil} arguemants.} \item{simil}{A numeric vector of similiarity values that represents the similiarity between the sentences represented by the IDs in \code{s1} and \code{s2}.} \item{threshold}{The minimum simil value a sentence pair must have to be represented in the graph where lexRank is calculated.} \item{n}{The number of sentences to return as the extractive summary. The function will return the top \code{n} lexRanked sentences. See \code{returnTies} for handling ties in lexRank.} \item{returnTies}{\code{TRUE} or \code{FALSE} indicating whether or not to return greater than \code{n} sentence IDs if there is a tie in lexRank. If \code{TRUE}, the returned number of sentences will not be limited to \code{n}, but rather will return every sentece with a top 3 score. If \code{FALSE}, the returned number of sentences will be \code{<=n}. Defaults to \code{TRUE}.} \item{usePageRank}{\code{TRUE} or \code{FALSE} indicating whether or not to use the page rank algorithm for ranking sentences. If \code{FALSE}, a sentences unweighted centrality will be used as the rank. Defaults to \code{TRUE}.} \item{damping}{The damping factor to be passed to page rank algorithm. Ignored if \code{usePageRank} is \code{FALSE}.} \item{continuous}{\code{TRUE} or \code{FALSE} indicating whether or not to use continuous LexRank. Only applies if \code{usePageRank==TRUE}. If \code{TRUE}, \code{threshold} will be ignored and lexRank will be computed using a weighted graph representation of the sentences. Defaults to \code{FALSE}.} } \value{ A 2 column dataframe with columns \code{sentenceId} and \code{value}. \code{sentenceId} contains the ids of the top \code{n} sentences in descending order by \code{value}. \code{value} contains page rank score (if \code{usePageRank==TRUE}) or degree centrality (if \code{usePageRank==FALSE}). } \description{ Compute LexRanks from sentence pair similarities using the page rank algorithm or degree centrality the methods used to compute lexRank are discussed in "LexRank: Graph-based Lexical Centrality as Salience in Text Summarization." } \examples{ lexRankFromSimil(s1=c("d1_1","d1_1","d1_2"), s2=c("d1_2","d2_1","d2_1"), simil=c(.01,.03,.5)) } \references{ \url{http://www.cs.cmu.edu/afs/cs/project/jair/pub/volume22/erkan04a-html/erkan04a.html} } lexRankr/man/sentenceTokenParse.Rd0000644000175100001440000000501313056025671016724 0ustar hornikusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/sentenceTokenParse.R \name{sentenceTokenParse} \alias{sentenceTokenParse} \title{Parse text into sentences and tokens} \usage{ sentenceTokenParse(text, docId = "create", removePunc = TRUE, removeNum = TRUE, toLower = TRUE, stemWords = TRUE, rmStopWords = TRUE) } \arguments{ \item{text}{A character vector of documents to be parsed into sentences and tokenized.} \item{docId}{A character vector of document Ids the same length as \code{text}. If \code{docId=="create"} document Ids will be created.} \item{removePunc}{\code{TRUE} or \code{FALSE} indicating whether or not to remove punctuation from \code{text} while tokenizing. If \code{TRUE}, puncuation will be removed. Defaults to \code{TRUE}.} \item{removeNum}{\code{TRUE} or \code{FALSE} indicating whether or not to remove numbers from \code{text} while tokenizing. If \code{TRUE}, numbers will be removed. Defaults to \code{TRUE}.} \item{toLower}{\code{TRUE} or \code{FALSE} indicating whether or not to coerce all of \code{text} to lowercase while tokenizing. If \code{TRUE}, \code{text} will be coerced to lowercase. Defaults to \code{TRUE}.} \item{stemWords}{\code{TRUE} or \code{FALSE} indicating whether or not to stem resulting tokens. If \code{TRUE}, the outputted tokens will be tokenized using \code{SnowballC::wordStem()}. Defaults to \code{TRUE}.} \item{rmStopWords}{\code{TRUE}, \code{FALSE}, or character vector of stopwords to remove from tokens. If \code{TRUE}, words in \code{tm::stopwords("SMART")} will be removed prior to stemming. If \code{FALSE}, no stopword removal will occur. If a character vector is passed, this vector will be used as the list of stopwords to be removed. Defaults to \code{TRUE}.} } \value{ A list of dataframes. The first element of the list returned is the \code{sentences} dataframe; this dataframe has columns \code{docId}, \code{sentenceId}, & \code{sentence} (the actual text of the sentence). The second element of the list returned is the \code{tokens} dataframe; this dataframe has columns \code{docId}, \code{sentenceId}, & \code{token} (the actual text of the token). } \description{ Parse a character vector of documents into into both sentences and a clean vector of tokens. The resulting output includes IDs for document and sentence for use in other \code{lexRank} functions. } \examples{ sentenceTokenParse(c("Bill is trying to earn a Ph.D.", "You have to have a 5.0 GPA."), docId=c("d1","d2")) } lexRankr/man/tokenize.Rd0000644000175100001440000000353213056025671014760 0ustar hornikusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/tokenize.R \name{tokenize} \alias{tokenize} \title{Tokenize a character vector Parse the elements of a character vector into a list of cleaned tokens.} \usage{ tokenize(text, removePunc = TRUE, removeNum = TRUE, toLower = TRUE, stemWords = TRUE, rmStopWords = TRUE) } \arguments{ \item{text}{The character vector to be tokenized} \item{removePunc}{\code{TRUE} or \code{FALSE} indicating whether or not to remove punctuation from \code{text}. If \code{TRUE}, puncuation will be removed. Defaults to \code{TRUE}.} \item{removeNum}{\code{TRUE} or \code{FALSE} indicating whether or not to remove numbers from \code{text}. If \code{TRUE}, numbers will be removed. Defaults to \code{TRUE}.} \item{toLower}{\code{TRUE} or \code{FALSE} indicating whether or not to coerce all of \code{text} to lowercase. If \code{TRUE}, \code{text} will be coerced to lowercase. Defaults to \code{TRUE}.} \item{stemWords}{\code{TRUE} or \code{FALSE} indicating whether or not to stem resulting tokens. If \code{TRUE}, the outputted tokens will be tokenized using \code{SnowballC::wordStem()}. Defaults to \code{TRUE}.} \item{rmStopWords}{\code{TRUE}, \code{FALSE}, or character vector of stopwords to remove. If \code{TRUE}, words in \code{tm::stopwords("SMART")} will be removed prior to stemming. If \code{FALSE}, no stopword removal will occur. If a character vector is passed, this vector will be used as the list of stopwords to be removed. Defaults to \code{TRUE}.} } \description{ Tokenize a character vector Parse the elements of a character vector into a list of cleaned tokens. } \examples{ tokenize("Mr. Feeny said the test would be on Sat. At least I'm 99.9\% sure that's what he said.") tokenize("Bill is trying to earn a Ph.D. in his field.", rmStopWords=FALSE) } lexRankr/man/lexRank.Rd0000644000175100001440000001110313056025671014525 0ustar hornikusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/lexRank.R \name{lexRank} \alias{lexRank} \title{Extractive text summarization with LexRank} \usage{ lexRank(text, docId = "create", threshold = 0.2, n = 3, returnTies = TRUE, usePageRank = TRUE, damping = 0.85, continuous = FALSE, sentencesAsDocs = FALSE, removePunc = TRUE, removeNum = TRUE, toLower = TRUE, stemWords = TRUE, rmStopWords = TRUE, Verbose = TRUE) } \arguments{ \item{text}{A character vector of documents to be cleaned and processed by the LexRank algorithm} \item{docId}{A vector of document IDs with length equal to the length of \code{text}. If \code{docId == "create"} then doc IDs will be created as an index from 1 to \code{n}, where \code{n} is the length of \code{text}.} \item{threshold}{The minimum simil value a sentence pair must have to be represented in the graph where lexRank is calculated.} \item{n}{The number of sentences to return as the extractive summary. The function will return the top \code{n} lexRanked sentences. See \code{returnTies} for handling ties in lexRank.} \item{returnTies}{\code{TRUE} or \code{FALSE} indicating whether or not to return greater than \code{n} sentence IDs if there is a tie in lexRank. If \code{TRUE}, the returned number of sentences will not be limited to \code{n}, but rather will return every sentece with a top 3 score. If \code{FALSE}, the returned number of sentences will be \code{<=n}. Defaults to \code{TRUE}.} \item{usePageRank}{\code{TRUE} or \code{FALSE} indicating whether or not to use the page rank algorithm for ranking sentences. If \code{FALSE}, a sentences unweighted centrality will be used as the rank. Defaults to \code{TRUE}.} \item{damping}{The damping factor to be passed to page rank algorithm. Ignored if \code{usePageRank} is \code{FALSE}.} \item{continuous}{\code{TRUE} or \code{FALSE} indicating whether or not to use continuous LexRank. Only applies if \code{usePageRank==TRUE}. If \code{TRUE}, \code{threshold} will be ignored and lexRank will be computed using a weighted graph representation of the sentences. Defaults to \code{FALSE}.} \item{sentencesAsDocs}{\code{TRUE} or \code{FALSE}, indicating whether or not to treat sentences as documents when calculating tfidf scores for similarity. If \code{TRUE}, inverse document frequency will be calculated as inverse sentence frequency (useful for single document extractive summarization).} \item{removePunc}{\code{TRUE} or \code{FALSE} indicating whether or not to remove punctuation from text while tokenizing. If \code{TRUE}, puncuation will be removed. Defaults to \code{TRUE}.} \item{removeNum}{\code{TRUE} or \code{FALSE} indicating whether or not to remove numbers from text while tokenizing. If \code{TRUE}, numbers will be removed. Defaults to \code{TRUE}.} \item{toLower}{\code{TRUE} or \code{FALSE} indicating whether or not to coerce all of text to lowercase while tokenizing. If \code{TRUE}, \code{text} will be coerced to lowercase. Defaults to \code{TRUE}.} \item{stemWords}{\code{TRUE} or \code{FALSE} indicating whether or not to stem resulting tokens. If \code{TRUE}, the outputted tokens will be tokenized using \code{SnowballC::wordStem()}. Defaults to \code{TRUE}.} \item{rmStopWords}{\code{TRUE}, \code{FALSE}, or character vector of stopwords to remove from tokens. If \code{TRUE}, words in \code{tm::stopwords("SMART")} will be removed prior to stemming. If \code{FALSE}, no stopword removal will occur. If a character vector is passed, this vector will be used as the list of stopwords to be removed. Defaults to \code{TRUE}.} \item{Verbose}{\code{TRUE} or \code{FALSE} indicating whether or not to \code{cat} progress messages to the console while running. Defaults to \code{TRUE}.} } \value{ A 2 column dataframe with columns \code{sentenceId} and \code{value}. \code{sentence} contains the ids of the top \code{n} sentences in descending order by \code{value}. \code{value} contains page rank score (if \code{usePageRank==TRUE}) or degree centrality (if \code{usePageRank==FALSE}). } \description{ Compute LexRanks from a vector of documents using the page rank algorithm or degree centrality the methods used to compute lexRank are discussed in "LexRank: Graph-based Lexical Centrality as Salience in Text Summarization." } \examples{ lexRank(c("This is a test.","Tests are fun.", "Do you think the exam will be hard?","Is an exam the same as a test?", "How many questions are going to be on the exam?")) } \references{ \url{http://www.cs.cmu.edu/afs/cs/project/jair/pub/volume22/erkan04a-html/erkan04a.html} } lexRankr/LICENSE0000644000175100001440000000005713056025670013071 0ustar hornikusersYEAR: 2016 COPYRIGHT HOLDER: Adam Spannbauer