Skip to content

GitLab

S
soneti
Commit 78fe4ac3 authored by Alberto Pascual's avatar Alberto Pascual
Browse files
Options

readthedocs updated, added docker images for soneti

parent 23566aca
Hide whitespace changes
Inline Side-by-side
Showing with 409 additions and 1692 deletions
.gitmodules 0 → 100644
View file @ 78fe4ac3
[submodule "readthedocs/gsicrawler"]
path = readthedocs/gsicrawler
url = http://lab.cluster.gsi.dit.upm.es/gsicrawler/gsicrawler.git
[submodule "readthedocs/senpy"]
path = readthedocs/senpy
url = http://lab.cluster.gsi.dit.upm.es/senpy/senpy.git
[submodule "readthedocs/orchestrator"]
path = readthedocs/orchestrator
url = http://lab.cluster.gsi.dit.upm.es/social/orchestrator.git
[submodule "gsicrawler"]
path = gsicrawler
url = http://lab.cluster.gsi.dit.upm.es/gsicrawler/gsicrawler.git
[submodule "senpy"]
path = senpy
url = http://lab.cluster.gsi.dit.upm.es/senpy/senpy.git
[submodule "orchestrator"]
path = orchestrator
url = http://lab.cluster.gsi.dit.upm.es/social/orchestrator.git
orchestrator @ 51cbed22
Subproject commit 51cbed2230fb278e54d646264578ca2874597f74
readthedocs/_build/html/.buildinfo
View file @ 78fe4ac3
# Sphinx build info version 1
# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
config: 014296aa2ea9b8791431c140fa457adc
config: c9afd29bd9b8ff4b7bfb2295cdd28edb
tags: 645f666f9bcd5a90fca523b33c5a78b7
readthedocs/_build/html/_sources/gsicrawler/gsicrawler/senpy/README.rst.txt deleted 100644 → 0
View file @ 23566aca
.. image:: img/header.png
:height: 6em
:target: http://demos.gsi.dit.upm.es/senpy
.. image:: https://travis-ci.org/gsi-upm/senpy.svg?branch=master
:target: https://travis-ci.org/gsi-upm/senpy
Senpy lets you create sentiment analysis web services easily, fast and using a well known API.
As a bonus, senpy services use semantic vocabularies (e.g. `NIF <http://persistence.uni-leipzig.org/nlp2rdf/>`_, `Marl <http://www.gsi.dit.upm.es/ontologies/marl>`_, `Onyx <http://www.gsi.dit.upm.es/ontologies/onyx>`_) and formats (turtle, JSON-LD, xml-rdf).
Have you ever wanted to turn your sentiment analysis algorithms into a service?
With senpy, now you can.
It provides all the tools so you just have to worry about improving your algorithms:
`See it in action. <http://demos.gsi.dit.upm.es/senpy>`_
Installation
------------
The stable version can be installed in three ways.
Through PIP
***********
.. code:: bash
pip install --user senpy
Alternatively, you can use the development version:
.. code:: bash
git clone git@github.com:gsi-upm/senpy
cd senpy
pip install --user .
If you want to install senpy globally, use sudo instead of the ``--user`` flag.
Docker Image
************
Build the image or use the pre-built one: ``docker run -ti -p 5000:5000 balkian/senpy --host 0.0.0.0 --default-plugins``.
To add custom plugins, add a volume and tell senpy where to find the plugins: ``docker run -ti -p 5000:5000 -v <PATH OF PLUGINS>:/plugins balkian/senpy --host 0.0.0.0 --default-plugins -f /plugins``
Usage
-----
However, the easiest and recommended way is to just use the command-line tool to load your plugins and launch the server.
.. code:: bash
senpy
or, alternatively:
.. code:: bash
python -m senpy
This will create a server with any modules found in the current path.
For more options, see the `--help` page.
Alternatively, you can use the modules included in senpy to build your own application.
Deploying on Heroku
-------------------
Use a free heroku instance to share your service with the world.
Just use the example Procfile in this repository, or build your own.
`DEMO on heroku <http://senpy.herokuapp.com>`_
For more information, check out the `documentation <http://senpy.readthedocs.org>`_.
------------------------------------------------------------------------------------
Acknowledgement
---------------
This development has been partially funded by the European Union through the MixedEmotions Project (project number H2020 655632), as part of the `RIA ICT 15 Big data and Open Data Innovation and take-up` programme.
.. image:: img/me.png
:target: http://mixedemotions-project.eu
:height: 100px
:alt: MixedEmotions Logo
.. image:: img/eu-flag.jpg
:height: 100px
:target: http://ec.europa.eu/research/participants/portal/desktop/en/opportunities/index.html
readthedocs/_build/html/_sources/gsicrawler/gsicrawler/senpy/docs/api.rst.txt deleted 100644 → 0
View file @ 23566aca
NIF API
=======
.. http:get:: /api
Basic endpoint for sentiment/emotion analysis.
**Example request**:
.. sourcecode:: http
GET /api?input=I%20love%20GSI HTTP/1.1
Host: localhost
Accept: application/json, text/javascript
**Example response**:
.. sourcecode:: http
HTTP/1.1 200 OK
Vary: Accept
Content-Type: text/javascript
{
"@context": [
"http://127.0.0.1/static/context.jsonld",
],
"analysis": [
{
"@id": "SentimentAnalysisExample",
"@type": "marl:SentimentAnalysis",
"dc:language": "en",
"marl:maxPolarityValue": 10.0,
"marl:minPolarityValue": 0.0
}
],
"domain": "wndomains:electronics",
"entries": [
{
"opinions": [
{
"prov:generatedBy": "SentimentAnalysisExample",
"marl:polarityValue": 7.8,
"marl:hasPolarity": "marl:Positive",
"marl:describesObject": "http://www.gsi.dit.upm.es",
}
],
"nif:isString": "I love GSI",
"strings": [
{
"nif:anchorOf": "GSI",
"nif:taIdentRef": "http://www.gsi.dit.upm.es"
}
]
}
]
}
:query i input: No default. Depends on informat and intype
:query f informat: one of `turtle` (default), `text`, `json-ld`
:query t intype: one of `direct` (default), `url`
:query o outformat: one of `turtle` (default), `text`, `json-ld`
:query p prefix: prefix for the URIs
:query algo algorithm: algorithm/plugin to use for the analysis. For a list of options, see :http:get:`/api/plugins`. If not provided, the default plugin will be used (:http:get:`/api/plugins/default`).
:reqheader Accept: the response content type depends on
:mailheader:`Accept` header
:resheader Content-Type: this depends on :mailheader:`Accept`
header of request
:statuscode 200: no error
:statuscode 404: service not found
.. http:post:: /api
The same as :http:get:`/api`.
.. http:get:: /api/plugins
Returns a list of installed plugins.
**Example request**:
.. sourcecode:: http
GET /api/plugins HTTP/1.1
Host: localhost
Accept: application/json, text/javascript
**Example response**:
.. sourcecode:: http
{
"@context": {
...
},
"sentiment140": {
"name": "sentiment140",
"is_activated": true,
"version": "0.1",
"extra_params": {
"@id": "extra_params_sentiment140_0.1",
"language": {
"required": false,
"@id": "lang_sentiment140",
"options": [
"es",
"en",
"auto"
],
"aliases": [
"language",
"l"
]
}
},
"@id": "sentiment140_0.1"
},
"rand": {
"name": "rand",
"is_activated": true,
"version": "0.1",
"extra_params": {
"@id": "extra_params_rand_0.1",
"language": {
"required": false,
"@id": "lang_rand",
"options": [
"es",
"en",
"auto"
],
"aliases": [
"language",
"l"
]
}
},
"@id": "rand_0.1"
}
}
.. http:get:: /api/plugins/<pluginname>
Returns the information of a specific plugin.
**Example request**:
.. sourcecode:: http
GET /api/plugins/rand HTTP/1.1
Host: localhost
Accept: application/json, text/javascript
**Example response**:
.. sourcecode:: http
{
"@id": "rand_0.1",
"extra_params": {
"@id": "extra_params_rand_0.1",
"language": {
"@id": "lang_rand",
"aliases": [
"language",
"l"
],
"options": [
"es",
"en",
"auto"
],
"required": false
}
},
"is_activated": true,
"name": "rand",
"version": "0.1"
}
.. http:get:: /api/plugins/default
Return the information about the default plugin.
.. http:get:: /api/plugins/<pluginname>/{de}activate
{De}activate a plugin.
**Example request**:
.. sourcecode:: http
GET /api/plugins/rand/deactivate HTTP/1.1
Host: localhost
Accept: application/json, text/javascript
**Example response**:
.. sourcecode:: http
{
"@context": {},
"message": "Ok"
}
readthedocs/_build/html/_sources/gsicrawler/gsicrawler/senpy/docs/index.rst.txt deleted 100644 → 0
View file @ 23566aca
.. Senpy documentation master file, created by
sphinx-quickstart on Tue Feb 24 08:57:32 2015.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to Senpy's documentation!
=================================
Contents:
.. toctree::
installation
usage
api
plugins
:maxdepth: 2
readthedocs/_build/html/_sources/gsicrawler/gsicrawler/senpy/docs/installation.rst.txt deleted 100644 → 0
View file @ 23566aca
Installation
------------
The stable version can be installed in three ways.
Through PIP
***********
.. code:: bash
pip install --user senpy
Alternatively, you can use the development version:
.. code:: bash
git clone git@github.com:gsi-upm/senpy
cd senpy
pip install --user .
If you want to install senpy globally, use sudo instead of the ``--user`` flag.
Docker Image
************
Build the image or use the pre-built one: ``docker run -ti -p 5000:5000 balkian/senpy --host 0.0.0.0 --default-plugins``.
To add custom plugins, add a volume and tell senpy where to find the plugins: ``docker run -ti -p 5000:5000 -v <PATH OF PLUGINS>:/plugins balkian/senpy --host 0.0.0.0 --default-plugins -f /plugins``
readthedocs/_build/html/_sources/gsicrawler/gsicrawler/senpy/docs/plugins.rst.txt deleted 100644 → 0
View file @ 23566aca
Developing new plugins
----------------------
Plugins Interface
=================
The basic methods in a plugin are:
* __init__
* activate: used to load memory-hungry resources
* deactivate: used to free up resources
* analyse: called in every user requests. It takes in the parameters supplied by a user and should return a senpy Response.
Plugins are loaded asynchronously, so don't worry if the activate method takes too long. The plugin will be marked as activated once it is finished executing the method.
F.A.Q.
======
If I'm using a classifier, where should I train it?
???????????????????????????????????????????????????
Training a classifier can be time time consuming. To avoid running the training unnecessarily, you can use ShelfMixin to store the classifier. For instance:
.. code:: python
from senpy.plugins import ShelfMixin, SenpyPlugin
class MyPlugin(ShelfMixin, SenpyPlugin):
def train(self):
''' Code to train the classifier
'''
# Here goes the code
# ...
return classifier
def activate(self):
if 'classifier' not in self.sh:
classifier = self.train()
self.sh['classifier'] = classifier
self.classifier = self.sh['classifier']
def deactivate(self):
self.close()
You can speficy a 'shelf_file' in your .senpy file. By default the ShelfMixin creates a file based on the plugin name and stores it in that plugin's folder.
Where can I find more code examples?
????????????????????????????????????
See: `<http://github.com/gsi-upm/senpy-plugins-community>`_.
readthedocs/_build/html/_sources/gsicrawler/gsicrawler/senpy/docs/usage.rst.txt deleted 100644 → 0
View file @ 23566aca
Usage
-----
The easiest and recommended way is to just use the command-line tool to load your plugins and launch the server.
.. code:: bash
senpy
Or, alternatively:
.. code:: bash
python -m senpy
This will create a server with any modules found in the current path.
For more options, see the `--help` page.
Alternatively, you can use the modules included in senpy to build your own application.
readthedocs/_build/html/_sources/index.rst.txt
View file @ 78fe4ac3
......@@ -3,7 +3,7 @@
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to soneti's documentation!
Welcome to Soneti's documentation!
==================================
.. toctree::
......@@ -11,7 +11,11 @@ Welcome to soneti's documentation!
:caption: Contents:
what-is-soneti
uses-soneti
installation
usecases
conventions
.. uses-soneti
Indices and tables
==================
......
readthedocs/_build/html/_sources/readthedocs-master-d7f61581e2d91d8567035fccca4c146ac4ed975c/docs/index.rst.txt deleted 100644 → 0
View file @ 23566aca
.. Undocumented documentation master file, created by
sphinx-quickstart on Mon Apr 24 16:53:33 2017.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to Undocumented's documentation!
========================================
This demo shows a picture.
Its only purpose is to show you how to document a project from scratch.
.. toctree::
:maxdepth: 2
:caption: Contents:
installation
usage
Indices and tables
==================
* :ref:`search`
readthedocs/_build/html/_sources/readthedocs-master-d7f61581e2d91d8567035fccca4c146ac4ed975c/docs/installation.rst.txt deleted 100644 → 0
View file @ 23566aca
Installing undocumented
=======================
To get the source code for this demo repository just `download the file <https://lab.cluster.gsi.dit.upm.es/docs/readthedocs/repository/archive.zip?ref=master>`_ or clone the repository with git:
.. code-block:: bash
git clone https://lab.cluster.gsi.dit.upm.es/docs/readthedocs/tree/master
readthedocs/_build/html/_sources/readthedocs-master-d7f61581e2d91d8567035fccca4c146ac4ed975c/docs/usage.rst.txt deleted 100644 → 0
View file @ 23566aca
Using undocumented
==================
Open your browser with ``firefox docs/_build/html/index.html`` and enjoy the beauty of our image:
.. image:: ../iamundocumented.jpg
readthedocs/_build/html/_sources/what-is-soneti.rst.txt
View file @ 78fe4ac3
==============
What is Soneti
==============
===============
What is Soneti?
===============
**Soneti** is a toolkit for **analyzing social software**, such as social networks (e.g. Twitter, Facebook, ...), blogs, YouTube, AppStores, etc..
**Soneti** is a toolkit for **analyzing social software**, such as social networks (e.g. Twitter, Facebook, ...), blogs, YouTube, Newspapers, AppStores, etc..
Its purpose is to provide tools to carry out automatic analysis about the contents and the users of this social software. Some of the applications we have developed are sentiment and emotion analysis for brand monitoring, radicalism analysis, personality analysis of youtubers, to name a few.
It obtains data from diffetent sources, in addition it enrich this obtained data by performing different types of automatic analysis. Finally, it allows us to visualize the data obtained in interactive dashboards.
Some of the applications we have developed are sentiment and emotion analysis for brand monitoring, radicalism analysis, personality analysis of youtubers, to name a few.
.. figure:: figures/gsi-social-network-analysis.jpg
:scale: 80%
Architecture
~~~~~~~~~~~~
Soneti has a modular architecture and we differentiate three main modules, as we can see in the image.
.. figure:: figures/soneti.png
:alt: Soneti overview
Someti is structured into four blocks:
* **Ingestion tool:** This module is called **GSICrawler** and is responsible for obtaining data from different sources, whether they are newspapers or social networks. For more information about this module or to add new data sources visit the documentation on http://gsicrawler.readthedocs.io
* **Social Software Analysis:** tools for data enrichment. This module uses **Senpy**: an analysis service, it allows us to obtain the sentiments, emotions or detect entities present in a certain text. To obtain more information about Senpy or to add new plugins for analysis visit the documentation on http://senpy.readthedocs.io
* **Query & Visualization**: tools for visualuazing and querying the results. We provide **Sefarad**, which enables the development of dashboards and includes a semantic query interface. It is based on Web Components and allows us to obtain an interactive overview of the data collected and enriched through the previous modules. Soneti provides a predefined dashboard if you are interested in getting more information and creating your own dashboards visit the Sefarad's documentation on http://sefarad.readthedocs.io
* **Ingestion tools:** tools for aggregating information from social networks. Two approaches can be followed using an API (e.g. Twitter, Facebook or Logstash) or Scraping (e.g. GSICrawler and Scrappy).
* **Social Software Analysis:** tools for analysing the incoming information. The core of this tools is a workflow engine (**Luigi**) which orchestrates the data pipelines, from ingestion to analysis and storing. The information is usually analyzed with **Senpy** and stored in **ElasticSearch**.
* **Social Simulation:** tools for simulating the behaviour of social networks. The tool **Soil** provides an agent-based simulator based on **NetworkX**.
* **Query & Visualization**: tools for visualuazing and querying the results. We provide **Sefarad**, which enables the development of dashboards and includes a semantic query interface. In addition, **Kibana** can be used for visual analytics.
\ No newline at end of file
Above all these modules we find the orchestrator Luigi, who is in charge of coordinating and establishing the relationships between all the modules.
\ No newline at end of file