Topic Models Applied to Online News and Reviews

Topic Models Applied to Online News and Reviews

>> LIN: Okay, hi. My name is Jimmy Lin. I work
here at Google and I’m happy to introduce Professor Alice Oh. She’s an assistant professor
of Computer Science at the Korea Advanced Institute of Science and Technology. Professor
Oh’s group does research in natural language processing, machine learning and human computer
interaction. And today, she’ll be talking about the work that her group is doing on
Topic Models related to Online News and Reviews.>>OH: Okay. I’m very glad to be here and
thank you, Jimmy, for hosting this talk. So, today I’m going to be talking about Topic
Models and how we apply that to online reviews and online news. And in doing so, I’m going
to be talking about Topic Models a little bit just to make sure that everybody knows
what Topic Models are, so it’ll be a brief introduction to it. And then I’ll go into
the details of our work. These are sort of two mini-talks. The topics are–the second
and the third items are related but they’re not quite the same. And these are from two
papers that my students and I recently submitted to Wisdom next year. So I keep my fingers
crossed. Okay, so let’s just dive into the main problem that we’re going to be talking
about. And, recently Google Books has announced this, right? I’m sure many of you read about
this somewhere that Google Books has counted at least 130 million books out there ever
written; and I’m sure there are more. So what we can see, and I’m probably preaching to
the choir here, is that there is a lot of text data out there to be understood, right?
So if you have 130 million books, we can ask this question, it’s a very simple question
when you just look at it, “What are the books about?” But it’s a very challenging problem,
right? So if you know anything about text processing you’ll agree with me that if we
have 130 million books and we’re trying to figure out what actually is in those books
that’s a very difficult question. Topic Models is one answer, one approach to getting at
that answer, okay? So the plate diagram up there is what you normally see when we talk
about Topic Models so I just put it up there. We’ll get back to the plate diagram in a little
bit. But Topic Models, the main purpose of them is to understand–sorry, to understand
and uncover the underlying semantic structure of text of your corpus, okay? So let’s look
at an example of what a Topic Model could do for you, right? So this is an article from
the New York Times a couple days ago, a few days ago, and the title is “Economic Slowdown
Catches Up With NASCAR.” And as you can see from the headlines it’s talking about the
NASCAR car racing and it’s also talking about the economic recession. And what Topic Models
do for you is it kind of discovers, it uncovers what the latent topics are in an article or
in a corpus, okay? So you can see in these three colors here, green, orange and a little,
I guess, purplish pink color, the three topics; three of the main topics you can see from
the article. So the green one is about NASCAR races and you can see sort of throughout the
document I’ve highlighted the words that are about that topic, okay? So, NASCAR races,
track, raceway, cars, et cetera. And then, the same thing with orange, is about economic
recession, so you would see–or it’s like sales, costs and so on. And the purple is
the general sports topics. So, [INDISTINCT] Topic Models and the–a Topic Model would
have, it–every document is made up of multiple topics and the title words in the document
are generated from those multiple topics, okay? So LDA, the Latent Dirichlet Allocation,
is one of the simplest Topic Models and it’s very widely used, so–and it’s a generative
model, which means that it tries to mimic what the writing process is, right? So it
tries to generate a document given the topics, okay? So let’s that’s how that works. So bear
with me if you are experts on LDA. I’m sure some of you are. So, here again we start with
the three topics, the NASCAR races, economic recession, and the general sports topic. And
when you have those topics and notice the topics are made up of words–and I’m just
showing you a subset of the words that have high probabilities in that topic. But actually
the topics are multinomials over the entire vocabulary. So, the NASCAR race topic, it
has–it gives high probabilities to those words but there other words in that topic
and they have small probabilities. Okay, so when you have these multinomials over words,
when you want to, say, generate three documents from these three topics, what you would do
is kind of produce or kind of guess these topic distributions of the documents that
you’re trying to generate. So, for example, the middle–the one in the middle, the writer
is thinking, you know, “I’m going to write mostly about the general sports topic and
then I’ll talk a little bit about maybe some of the other topics,” okay? That’s what the
big purple bar means. Okay, so when you have those topic distributions, then what you can
do is–oops, I’m going backwards. Okay, so from, say, the bottom topic distribution,
you can generate the words according to that distribution, okay? So, since we have a lot
of the green, you would see many green words popping up there, okay? And then the same
thing you would do for the other documents. So that’s the generative process of an LDA.
Okay, so let’s look at it from the plate diagram perspective. So here up on the left is the
general–is the widely used plate diagram for this. And what you can see here is the
phi’s, okay, next to the beta there, are the topics, which are the multinomials over the
vocabulary. And then up there up to the right corner over there are the thetas which are
the topic distributions, okay? And then from those topic distributions if you want to generate
one document you would generate sort of the set of rectangles I have over there which
are the words or which are the topics of the words that you’re going to write in your document,
okay? So in this example, I’m looking at the first topic distribution and I’ve sort of
picked out the topics that I want to write about. And then, when you have those topics
then you can look up the multinomial topics over here to generate, to actually come up
with the words, okay? According–so, for example, the first word you’re going to write is in
orange, so you’re going to come here and look at the orange topic and say, “Okay, these
words have high probabilities in these topics I’m going to pick one of those words,” okay?
So that’s what you get in the actual–the rectangle down below, where you actually have
the words in your document, okay, and that’s how the document is generated. Okay, but in
reality, what you have is the–you only observe the words. So these are the documents in your
corpus, okay? So, where this is one document in your corpus and all others like the topic
distributions, the topic themselves, they’re latent. We don’t know them. But the purpose
of fitting the model then is to come up with those topic distributions and the topics,
okay? So they–all those others must be discovered by the model. Okay, so that’s what you do
when you, you know, fit an LDA. Okay, so what does an output of an LDA look like? They look
something like this. There are some other outputs that LDA gives you but one of the
major outputs of LDA is these multinomials over words which are the topics. So the NASCAR
topic, for example, has those words with high probabilities. What I put at the very bottom
row is to let you know that actually every topic has every word in the vocabulary. It’s
just that some like the money word here in the NASCAR topic has very low probability
of it. In actuality it’s probably much lower than that, okay? So that’s what the topics
look like. So if we go back to the question I posed earlier, if you have 130 million books
and you want to answer the question, “What are the books about?” then you can imagine
you can sort of feed these books into an LDA to discover topics, right? So if you represent
one book as one document and you run it over 130 million of them, you can discover the
underlying topics, the underlying semantic structure of your corpus. Let’s look at a
smaller problem since it’s very hard to run LDA on 130 million documents. But if we have
news articles and we have about 200,000 of them over the last 12 months then we can ask
the question, “What are the news articles about?” And this is something that we can
try to solve. And one difference here is that time is a very important dimension here, right?
Because news is inherently sequential and temporal and you want to know what happened
when and how long did it last and so on, okay? So we need something that considers time.
Okay, so we proposed what we call Topic Chains, which is–has the main purpose of uncovering
the underlying semantic structure of a sequential corpus of news, okay? And this is work that
my student did mostly and so if you have any detailed questions about it you can send them
an email, although I’ll try to answer most of them. By the way, if you have any questions,
feel free to raise your hand and ask. Okay, so let’s look at what we have, what kind of
tools that we have available to us now. So if you look at the New York Times, if you
kind of scrolled down to the bottom half of the page, this is what you get. You get a
quick news-at-a-glance type of a thing, right? So you can look at the New York Times, the
front page of that website and kind of figure out what’s been going on in the last couple
of days, okay, in terms of technology, the world, business, arts and so on. And this
is a very good view and I love to look at it. But this, as you can imagine, takes a
lot of intelligence and a lot of work, right? So this is a product of, you know, intelligent
New York Times editors out there who are trying–who are putting this together. And plus, it doesn’t
have the dimension of time because this is a snapshot of the news, right? So here, at
Google, somebody has made this really interesting tool it’s called Google News Timeline. It’s
still in the Google Labs, so you may not–may or may not know. But this is where you can
look at the sequential issues and events, right? So, right now, it’s showing the monthly
view. So you can see what was the more–most important news in March of 2009 and so on.
And you can search, too. So if you search for a certain keyword then you would get articles
that are about that keyword, right, and you can look at the weekly view and the yearly
view as well, I think, and the daily, of course. So this is pretty cool but I think that here
we still have some questions that are unresolved. For example, if you have an article, say,
there’s an important article in March of 2009, are there similar articles that follow that
are talking about the same thing in April of 2009, perhaps a couple or a few months
later, right? And if there are similar articles talking about the same topic over a long period
of time, how long is that period of time? How long did that topic last, right? And if
it’s a long-lasting topic, then is it part of a general sort of professional topic like
the U.S. economy or was it part of a long, running sort of event or issue such as the
H1N1 issue, right? Or was it part of a very short, temporary topic such as the death of
Michael Jackson, okay? So we would like to know those things when we look at the articles,
but at least what we saw in the previous slide didn’t really show that. And if it’s a general,
sort of long-running topic like the H1N1, for example, the topic itself kind of evolves
through the nine months or how many–however many months that it lasts, okay? First, it
was talking about the outbreak, perhaps, maybe it was talking about travel restrictions and
then vaccinations that’s–schools and so on, okay? So we would like to see how the same
topic evolves through time. So, what we propose is something like this. This is a part of
our results, is that you can look at several months of news and kind of look at the topics
and how they’re clustered together in what we call Topic Chains. So you–here you see
that there’s a Topic Chain about labor unions, education, the War in Afghanistan, the swine
flu and so on. And then you would see some events like there was a terror in Hong Kong
or something like that and the death of Michael Jackson. So we produced something like this
where you can see the general perpetual topics, you can see the long-running “It” topics and
then you can see sort of the temporary events that happened. So, this is what we call the
Topic Chains and this is the plan that we had to do something like that, right? So what
we did is we took a bunch of articles over a bunch of months and we divide the corpus
into time slices and we just chose time slice of 10 days each, okay? And for each time slice
you would have a bunch of articles, right, and we can find the topics using just the
simple LDA. And when you have the topics from the LDA then you can try to match them up
to see which topics are similar, okay? And when you have the similar topics, you can
sort of link them up into Topic Chains. And once you–yeah?
>>At what similarity are topics based on?>>OH: Yes, so I’ll talk about similarity
metrics in the next slide, I think, or a couple of slides. And then once we have the Topic
Chains we can identify which are the long topics, which are the short topics, and within
the long topics we can sort of see what the topic evolution looks like. Okay, so let me
talk now about sort of each of those steps; except for the first one because that one
is trivial. So we worked with the corpus of nine months of news in Korea. So we took the
websites of three major newspapers and collected documents and articles from all of them. The
corpus looks like that, 130,000 articles, 140,000 unique words, named entities, and
we chose 50 as the top–the number of topics per each time slice for a total of 1,400 topics.
And let me just show you the results of the LDA; finding topics using LDA. And there’s,
since there’s 1,400 of them, I can’t really show you them all, but I’m just showing you
four examples of how the topics turned out pretty good. The first one you can see is
about sports. The second one is about business and then about smartphones and technology.
And the last thing, the last topic is about Academia. So when we have those topics, we
can construct Topic Chains like this where you look for similar topics within certain
window size. And we also–I’ll show you an experiment that we did with increasing or
decreasing the window size and what happens there. But what it means is do you look at
only the time slice before or how many time slices do you go back to find the similar
topics? Okay, so here comes the answer to that question, measuring similarity. This
is kind of an important issue because the major thing about Topic Chains is that we’re
finding similar topics, right? So, remember, the topics look like those, which are multinomials
over words. And so you can imagine various ways to measure similarity and within the
Topping modeling research community people have used most of these metrics. Most notably,
they usually use KL divergence or cosine similar–cosine, yeah, cosine similarity. And I’ve kind of
categorized these six–or is it six or five–six similarity metrics by how each metric looks
at the topics, okay? So the first thing, you can look at–or I said that our topic is multinomial
over the vocabulary, right? So if you have two probability distributions and you want
to measure the distance, then KL divergence is the answer, right, or JS divergence, which
is the symmetric version of KL divergence, okay? Or you can look at a topic as a vector
where each dimension is a probability of the word in the topic. So if you take that view
then you can use cosine similarity because to measure the distance between two vectors,
right? Or you can use Kendall’s Tau if you look at a topic as a list of ranked words,
a ranked list of words. So if we just ignore the probabilities but just look at, you know,
NASCAR is the first rank and so on then we can use Kendall’s Tau or DCG which is used
I guess a lot in information retrieval. And then lastly, if you look at only the subset
of words that have top–high probabilities, then we can look at the intersection and unions
of sets, which we can measure with Jaccard’s coefficient. So we wanted to test these metrics
to see which would be the most or the best performing similarity metric, okay? So what
we did is this, we computed the log likelihood of data of the corpus, given the topics that
LDA found. What that means is if you have a small negative log likelihood of data given
the topics, then that means your topics are explaining your corpus very well, okay? So
the higher the value, there’s sort of a mismatch between your topics and your corpus, okay?
So it’s kind of like perplexity, too. So what we did is we took an original set of 50 topics
that LDA found for each–for one time slice and replaced five of those topics with similar
topics that are found by each of the metrics, each of the six metrics, okay? So, for example,
you know, if KL divergence says, among these 50 topics and then another set of 50 topics
in the next time slice these five or the most similar pairs, then we replace the topics
from the second time slice and kind of put them in the first time slice. So you would
have the 45 of the original topics plus five new ones that KL says is most similar or most
similar, okay? So then, when we compute the log likelihood of the modified topics or the
log likelihood of the data given the modified set of topics, then we can see which of the
similarity metrics found the most similar topics, okay? So, as I said before, KL divergence
and cosine similarity are most often used similarity metrics and we found that JS divergence
actually performs a little bit better. And the asterisks next to the metrics mean that
there is a significant difference, statistically significant difference, between that metric
and JS divergence. And Jaccard’s coefficient performs pretty well too, but we didn’t use
that because you have to have this parameter. There’s a parameter that we had to set and
we thought that that’s probably not as general as just using JS divergence with no parameter.
So, that’s what we chose to use as our similarity metric for constructing the Topic Chains.
And let’s now talk about the size of the window, the size of the sliding window. So if we can
take the [INDISTINCT] assumption and just look at one time slice to find a similar topics
but then you wouldn’t find that case of, sort of the long arrow over there, where a topic
was kind of an important issue for awhile and then it kind of disappears for a few weeks
and then comes back again. So, we didn’t want to miss that similarity chain there. So, this
is what we did as the experiment to see how the sliding window size affects the resulting
Topic Chains. So, up at the very top, is a set of Topic Chains found when we use the
sliding window of size one looking back just one time cycle, and then at the bottom is
the sliding window of size six. So, it’s kind of an obvious result, but you can see when
you’re looking back only one time slice then the Topic Chains are kind of fragmented and
they’re kind of dispersed all over the place. But as you increase the window size, then
the Topic Chains over there, that had sort of a little gap at the middle–in the middle
and then continued a few weeks later, they kind of merged together, right. So the Topic
Chains has become larger, longer, and you would find these pretty large Topic Chains
at the bottom. What’s interesting is, if you look the middle one, where there are two major
ones and then they come together at the size of five, those Topic Chains are about technology
and business. So one of them is about the technology itself, sort of manufacturing,
research, development, type of topics and then the other one is sort of the business
side of the technology, okay? So, you can see that they kind of merge together at the
window size of five, and it’s kind of hard to interpret that in terms of what it means
for the user, right? So if the user wants those to be kind of separately separate then
that’s probably what we should do. But if you want them to be sort of in this same Topic
Chain then you might want to go with the larger window size. But, in general, as you increase
the size of the sliding window, the Topic Chains tend to become more abstract. So at
the end, you would have something that’s similar to like the sections in your newspaper, right?
So, so business, life and, you know, culture, and, you know, the world news, and so on,
right; whereas sort of in the middle you would have sort of more concrete topics. Okay, so
that’s what that shows. Let’s look a little bit closer at the chains themselves, what
they mean. So if we look at the long chains, for example, the swine flu chain, right, you
want to know more than just that there was this big chain of swine flu and we can see
that, you know, in 2009 it was kind of a big issue for most of the year, we want to know
how that topic actually changed, okay? So as I talked about before, first it was talking
about the outbreak and then vaccinations and so on, right? So we call those focus shifts.
So within a Topic Chain, we can look at how the focus shifts in the chain. And I apologize
for the small font. It’s really hard to see. But this Topic Chain is about automobile industry,
okay? So, I’ll just read you–on topic number one has the tab words automobiles, Vietnam,
Kia Motors, vehicle and sales. And topic number three, which is right below that, is develop,
technology, automobile, investment, and industry, okay? And then the other topics are pretty
similar to that. So you really can’t tell what is going on just by looking at the topics
themselves, okay? So what we wanted to do is look at words that changed the most between
two similar topics. So, what happened between this topic and the next topic? And if we look
at the words that are not common but are most different among the two topics, then you can
sort of figure out what’s been going on. And on top of that, we looked at just the named
entities. Named entities are things like names of organizations, names of people, sort of
specific things like that, because a lot of the news, a lot of the events that happened
in the news are about specific people or organizations and so on, okay? So coming–going from number
one to number three, again, when we do that the–excuse me–the named entities that we
find are green, solar, Japan, energy, and what’s the last one–and carbon, yeah. So
that tells you that there was something going on in the second–so those are the words that
changed–that increased the most in probability from topic number one to topic number three.
And we–if we just go back and look at the headlines, you see that there was–there were
a few headlines that are talking about Japanese carmakers like Toyota coming up with solar
powered cars, okay? So, if we look at this sort of closed–close up view of the Topic
Chains and the named entities that changed, then we can have a much deeper understanding
of the evolution of the topics. Okay, now, let’s look at the short chains and this is
pretty interesting. Here, every line is a topic and the left column is the date. So,
0P 07 means the first 10 days of July in 2009, there was–there was a missile launched. There
was a discussion over the North Korea missile launch. And then the next line talks about
the death of Michael Jackson and then some milk scandal and then some heightened–a topic
about heightened security at the end of the year and then some romance over, you know,
entertainment people. In April of–April is when Korea has Arbor Day, so talking about
trees and stuff like that. And then the last topic is kind of interesting. Obama, Republicans,
Jeju Island is an island in Korea that’s used for resorts and playing golf and you see golf
and Tiger Woods. But, I don’t know if Obama went there to play golf or not, with Tiger
Woods maybe, but what we can interpret that is–that LDA found a topic that’s kind of
not about a single topic and LDA often does that. If you’ve ever run LDA or any other
Topic Models–oops–you’ll find many of–or some of the results, some of the topics that
you find are not really coherent. So, anyway, these are short Topic Chains which means they’re
like two or three topics or even one, two, or three topics, and they represent mostly
temporal events, temporal issues or they could be about incoherent topics. And you can kind
of see how, if it’s a coherent topic, then it would more easily find similar topics in
the next time windows, right? Okay, so that’s actually the end of the Topic Chains part
of the talk. How am I doing on time? Okay. I’ll go quickly over the next topic. So, we
propose Topic Chains, which is a framework based on very simple LDA to understand what’s
going on in the news corpus. Okay, now, let me switch gears and talk about sentiment and
aspects and reviews. So, the model is called Aspect Sentiment Unification Model and its
main purpose is to uncover the structure of aspects and sentiments in a review, okay?
And this is another student of mine who worked on this mostly. And the promise is this, if
you go to Amazon–this is a review of a digital camera. It’s a very long review. It’s like
a–it’s like a conference paper almost and this is actually not the end, there’s more.
But it’s a very, you know, detailed review. He talks about or this user talks about a
lot of good things and bad things about this camera and we want to do something like this,
right? Amazon does sort of aspect or attribute based sentiment analysis of the review. So,
in addition to the general, how many stars did this camera get? It also gives you how
many stars for the picture quality and so on. The way Amazon does it–I don’t know exactly
how they do it, but I noticed that–so this is a camera with lots and lots of reviews
like 300 reviews or so. And then there’s–the same Canon digital camera, certain other models
which have very few reviews, and for those we actually don’t have these attributes. So
it looks like there’s some manual work and some automated way of looking at what the
attributes are. And we call those attributes aspects, and there are things like this; this
thing is small and it’s light, starts up and turns off fast, the low light performance
is best. And so these are actual sentences from the reviews. And the sentiment is something
like this. The words highlighted in pink are the ones that carry sentiment for this–for
each sentence. Okay. So let’s look a little bit closely at what these sentiment words
are, okay? Some of them are general sort of effective words that express emotion like
love; “I love this,” “I’m satisfied,” “I’m dissatisfied,” “I’m disappointed,” okay? And
then, some of the other ones are general sentiment words like, “Best, excellent, bad.” There–they
evaluate the quality of something, but they’re just general. If something is best, then it’s
best no matter if it’s a coffeemaker or a chair, right? And then there are aspect specific
evaluative words. And this is a little more fine-grained than domain specific evaluative
words. So let me show you what I mean. In the camera domain, okay, if you say, “This
camera is small,” it’s probably a good thing. “The LCD is small,” it’s probably a bad thing,
right? If you’re in the restaurant domain, “The beer was cold,” is good. “Pizza was cold”
is bad. And, “The wine list is long,” is good and “The wait is long,” is bad. So, beyond
the domain, right, we need to go sort of down to each aspect of the review and say whether
the sentiment word there expresses positive or negative sentiment. Okay. So this is the
problem that we’re trying to solve. Okay, we’re trying to discover the aspects automatically
as well as the sentiment and the words that carry the sentiment. So to do that we made
two models, one is called “Sentence LDA,” the other is called “Aspect Sentiment Unification
Model.” And we worked with two types of corpura. The first one was Amazon reviews and we took
seven product categories, including digital cameras, coffeemakers, I think, heaters and
things like that. They’re just pretty different electronic products–oops. And we also looked
at the Yelp restaurant reviews over four cities and 328 restaurants. And on average, each
review had 12 sentences. And our observation starts again with the same set of sentences.
What we noticed here is that for many of the sentences in the reviews, one sentence describes
only one aspect, okay? And this is different from the general LDA assumption which is that
each word in the corpus, each word in the document, represents or is generated from
one topic or if you applied it to aspects, one aspect, okay? So we wanted to make this
sentence LDA. If you noticed, the only difference is the box around the W circle, okay? So what
that means is the words–and N is the number of words in your documents, so each word is
generated–but Z, which are the topics, are over M, which is the number of sentences,
okay? So we’re saying there are only M aspects in that document, which is the number of sentences
in that review, okay? And each sentence has one topic or one aspect, okay? So that’s the
basic difference between LDA and SLDA. And what we found is that when we run SLDA over
our data–so this is–oh, this–yeah, the results from both the Amazon reviews, and
the last one actually is from the restaurant review. So, remember, we ran SLDA over all
sort of seven categories of electronics reviews and we get these aspects. They are similar
to what we saw earlier in the Amazon attribute categories, okay? So the portability, quality
of photo and ease of use, those are the three–and the camera product. And then if you look at
the laptop reviews the first one is about software and OS and then the second one is
about hardware and so on. And some–and what we found when we compared the results of LDA
versus SLDA is that LDA or SLDA was finding more product specific aspects, okay? For example,
the last one, liquors category, liquors topic or aspect was not found by LDA. Instead, the
words like beer and wine and martini was actually one of the top words, too. They were kind
of spread out over different topics, like wine was maybe with the Italian food aspect
and so on. So, I think it’s important to notice that SLDA, because of that one difference
in the assumption it makes, finds better product specific aspects, details of the reviews.
Okay, we then took SLDA and extended it to form a joint model over aspects and sentiments,
okay? So the right side of the model, which has the gamma, the pi, and the S, so S is
the sentiment and you can see that word is now generated from a pair of sentiment and
aspect, okay? So, with this joint model then, if we run it over the corpus, without using
the labels, any labels of the corpus, of the documents, we can automatically discover the
aspects and the sentiments, okay? But we do use seed words. We took Turney’s Paradigm
words because they’re kind of generic paradigm sort of sentiment words that a lot of people
use, like good and nice, and bad and nasty and then–so that was one set of seed words
we used. We also augmented the paradigm words a little bit with other sort of general sentiment
words that we found from the corpus, okay? So–and what we do with this–with these sentiment
words that are a little bit different from other prior work in this joint modeling of
sentiment and aspect is that we build the seed words right into the model by playing
with the priors of the LDA, okay? So setting asymmetric priors and initializing Gibbs sampling,
which is an inference algorithm to kind of play with the seed words, okay? And I’ll–let
me explain that a little bit better here, although I didn’t even talk about Gibbs sampling,
so if you want to explain that, we can talk later after the talk. So beta is the prior
for the Dirichlet Distribution over the phi’s, okay? And what that means is, do we start
with a uniform distribution of betas, which means that every distribution is equally likely?
If we play with the betas and do asymmetric priors, then we’re saying some of the distributions
are more likely than others. Okay, so what we do–what we did with SLDA is we just used
the uniform priors. What we do with the betas here is we set zero–beta to zero, for any
negative sentiment seed words in the other, in the opposite–in the positive sentiments,
okay? That means–and we do vice-versa for the negative–for the positive sentiment seed
words, okay? That means if you have a positive seed word like “good,” then it’s not going
to be assigned a non-zero probability in a negative sentiment–negative aspect sentiment,
okay? And also, we start Gibbs sampling, we initialize a Gibbs sampling by setting the
positive seed words to have positive sentiment and the negative seed words to have negative
sentiment. So that’s opposed to randomly assigning sentiment, which is what we usually do for
Gibbs sampling, okay? So the combination of those two makes the seed words kind of right
into the model without fidgeting anymore with the words themselves. Okay. So these are senti
aspects discovered by ASUM, as we call the model, and these–so, every multinomial now
is–so every word in the multinomial is generated either by the sentiment or by the aspect or
they’re actually jointly by the pair sentiment and aspect, okay? So interesting results here,
like the meat senti aspects here–meat positive and meat negative, the meat aspect was not
found by SLDA. So what this tells you is that for some senti aspects, if there’s strong
sort of sentiment correlated with that aspect, then it comes out better with the ASUM model
than it does for just SLDA without sentiments built-in. So in original SLDA, what happens
is the meat aspect is kind of scattered around again in, you know, in pizza, in burger steak,
right? Those aspects have meat words in them. But because we’ve forced kind of the sentiment
to play a bigger role in finding the aspects, we see aspects like that. And an interesting
case that we see with payment is that we only get a negative aspect for payment. We don’t
get a positive senti aspect for payment. It’s the same thing with parking, too. What that
tells us, just an interesting bit, is that people complain about payment not being able
to use their credit card or they complain about parking situations. But if they have
some satisfaction with it, they don’t really write it in the reviews. Okay. And the yummy
aspect is kind of funny; too, with the last word is funny, right? So that aspect is something
that LDA doesn’t really find. Okay. So let me go on. So what we can do with these topics,
okay, the words in the topics is then we can try to figure out which are the sentiment
words and which are the aspect words, right? So if we have the two meat senti aspect words,
we can look at the words that appear common across the two sentiments like meat and–I
don’t know what else, sauce, I think, and we say, “Those are the common sort of aspect
words for the meat category,” okay? Whereas, things like “crispy” or “blend” are the sentiment
carrying words for that aspect, okay? So those are the aspect specific sentiment words. And
so, what we do is we align the senti aspects with the similar aspects again and then we
look at the positive aspects, the negative aspects, and we look at the common words and
the words that have a lot of difference in them to figure out things like this. The screen
aspect, the words, the common words are like screen, glossy LCD and then the sentiment
words are like bright, clear. Those are the positive words and, like, reflect, glare–MacBook,
obviously, it doesn’t have a good screen. So–or apparently. I don’t agree with that
personally, but anyway, so that’s what we can do with those topics. Okay, let’s look
at some other results of this. So here is a result that shows you that sentiment classification
per sentence is done pretty well. So we–so these are two reviews. The first one is about
a coffeemaker and then the second one is about a restaurant. And you can see in green, those
are the positive sentiment sentences. So that’s what the model found as the positive sentences
and the ones in pink are the–where the model found them to be negative. And, of course,
I’m going to show you good examples. But most of the examples are pretty good, okay? So,
another set of results we can do–we can look at is, how well are the aspects assigned to
the sentences, right? So these are four different reviews where the same aspect was found. The
aspect–the senti aspect of parking and the negative sentiment, and you can see that parking
is only validated for three hours and so on. So those are–and these came out pretty well,
right? Here’s another example. Some of these things like very convenient, how the model
find–found that to be coffeemaker easy? It could be some–I don’t know. It’s probably
just because convenient is up there as one of the top probability words for that senti
aspect. Some of the other shorter sentences this model has trouble with because there’s
not enough clue, okay? But–and I do like to show some of the bad examples as well.
So the second one, it took us several uses to understand how much coffee to use. That’s
obviously not a positive sentiment, but the model classified it as that. But, you know,
one out of five is not bad, right? Okay, so those are the senti aspects assigned to sentences
that we can see. Oh, and the last one, I put in there to show you that our assumption that
one sentence carries one aspect may not always be true, right? So the last sentence is talking
about how nice it looks and how easy it is to use. But you can kind of say, you know,
“Are they the same thing or not?” I don’t think they’re really the same thing. Nice
looking is probably–or it should be another aspect, sort of the design aspect of the product
and then the ease of use is the usability of the product, right? So we do see a lot
of sentences in our corpus that do not validate our assumption, that of one sentence equals
one aspect or even one sentence is one sentiment. But–so that’s future work for us, right,
to deal with sentences like that. Okay, so all of these that I’ve–the results that I’ve
showed you, because of the way Topic Models, you know, they produce these topics, it’s
really hard to evaluate them. There’s really no good way to quantitatively evaluate the
aspect. So we can’t ask users to go through 20,000 reviews and find all the aspects and
kind of compare them against our results, right, or the sentiments. So the sentiment
would be a little bit easier to do. So what we do with the sentiment actually, is we quantitatively
measured how sentiment classification is done against other generative models that jointly
model sentiment and aspect together. And so, this is–I have to tell you, though, so these–our
model as well these other models, JST and TSM are the two models that we’re comparing
against, they’re all not designed for sentiment classification per se. So they’re all trying
to discover aspects and sentiments together and come up with these sentiment words and
so on. They’re not, you know, models to do classification and neither is ours. But we
put this experiment in there to show at least that sentiments are found well, okay? So let
me explain the different things. ASUM, the blue, is our model with the regular paradigm
words. I think there’s like a dozen of the paradigm words that I showed you. ASUM Plus
is the paradigm plus words, the augmented list of words, and then GST Plus and TSM Plus
we also–they also use seed words, so we use the same set of paradigm plus words. And we
implemented those two models and rank classification over our own corpus to see how they–how well
they perform. So you can see the red line, which is ASUM Plus, performs the best. The
next one is the blue with–so, ASUM without the paradigm plus words and then the other
models don’t perform as well on the classification task. So, just to tell you, these models are
pretty similar to ours. They both don’t have the one sentence, one aspect assumption built-in.
They don’t use the seed words right into the model they do something else with the seed
words. So those are sort of the main differences I would say between those two models and our
model. Okay, so let me just wrap up. I think time is probably up, too. I just talked about
SLDA and ASUM, which are the two models, extensions of the basic LDA to discover sentiment and
aspect together. And we discovered that specific aspects that we found were pretty well aligned
with the details of the reviews that people actually write and we can, by looking at the
topics and the words within the topics, we can learn aspect-specific sentiment words.
And lastly, we just tested with sentiment classification and found that it performs
pretty well, okay? So just to wrap up now really, Topic Chains and ASUM are the two
things that I’ve talked about and they both work with LDA, right? So they’re on different
domains, one is on the news domain and then the other one is in the reviews domain. We’re
trying to do the same thing. We’re trying to uncover what is latent, what is the hidden
semantic structure within the news corpus and within the reviews corpus. So if you’re
interested in our work, further discussions with me and my students, please send us email
or you can look at our website to see what the latest things are going on. Okay. Thank
you. Questions?>>Could you comment on what you discovered
when you ran these two books? Like you were mentioning the…
>>OH: Yeah.>>…130 million books. These books are a
lot different than…>>OH: Yeah, yeah. So we didn’t–we haven’t
done that.>>You haven’t done that? Oh, okay.
>>OH: Yeah. Well, I should ask Google Books to do that for me. I don’t–I can’t imagine
what the results would be. One thing about LDA, though is that–and Topic Models in general,
is that they are very computationally expensive. As you can imagine, right? If–as the document
size grows large, the number of vocabulary–unique vocabulary grows large and you’re doing Gibbs
sampling over all your vocabulary at each iteration, and you have to do thousands of
iterations to converge. So the inference part is difficult. And we would like to maybe use
Google and, you know, use distributed computing and all that to figure out how to do that.
Oh, question?>>So the aspects you found usually it seems
not to the ones defined by users, like the camera. Is there any way you can specify,
“Okay, I like to find those aspects specified by users?”
>>OH: Yeah, that’s a good question. So we are thinking about it. We haven’t done anything.
I don’t know how to do it. Sort of like the sentiment seed words you can have maybe seed
words for aspects too, to say, “I want to find these aspects.” Good question and a good
idea for an extension of this work. Okay. Thank you, again.


6 thoughts on “Topic Models Applied to Online News and Reviews”

Leave a Reply

Your email address will not be published. Required fields are marked *