plugins.rst 8.69 KB
Newer Older
1 2
Developing new plugins
----------------------
Ignacio Corcuera's avatar
Ignacio Corcuera committed
3
Each plugin represents a different analysis process.There are two types of files that are needed by senpy for loading a plugin:
4

Ignacio Corcuera's avatar
Ignacio Corcuera committed
5 6 7
- Definition file, has the ".senpy" extension.
- Code file, is a python file.

J. Fernando Sánchez's avatar
J. Fernando Sánchez committed
8 9 10 11 12 13
This separation will allow us to deploy plugins that use the same code but employ different parameters.
For instance, one could use the same classifier and processing in several plugins, but train with different datasets.
This scenario is particularly useful for evaluation purposes.

The only limitation is that the name of each plugin needs to be unique.

Ignacio Corcuera's avatar
Ignacio Corcuera committed
14 15 16
Plugins Definitions
===================

J. Fernando Sánchez's avatar
J. Fernando Sánchez committed
17 18
The definition file contains all the attributes of the plugin, and can be written in YAML or JSON.
The most important attributes are:
Ignacio Corcuera's avatar
Ignacio Corcuera committed
19

J. Fernando Sánchez's avatar
J. Fernando Sánchez committed
20 21 22 23
* **name**: unique name that senpy will use internally to identify the plugin.
* **module**: indicates the module that contains the plugin code, which will be automatically loaded by senpy.
* **version**
* extra_params: used to specify parameters that the plugin accepts that are not already part of the senpy API. Those parameters may be required, and have aliased names. For instance:
Ignacio Corcuera's avatar
Ignacio Corcuera committed
24

J. Fernando Sánchez's avatar
J. Fernando Sánchez committed
25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52
  .. code:: yaml

            extra_params:
                hello_param:
                    aliases: # required
                        - hello_param
                        - hello
                    required: true
                    default: Hi you
                    values:
                        - Hi you
                        - Hello y'all
                        - Howdy

  Parameter validation will fail if a required parameter without a default has not been provided, or if the definition includes a set of values and the provided one does not match one of them.


A complete example:

.. code:: yaml
          
          name: <Name of the plugin>
          module: <Python file>
          version: 0.1

And the json equivalent:

.. code:: json
Ignacio Corcuera's avatar
Ignacio Corcuera committed
53 54

          {
J. Fernando Sánchez's avatar
J. Fernando Sánchez committed
55 56 57
            "name": "<Name of the plugin>",
            "module": "<Python file>",
            "version": "0.1"
Ignacio Corcuera's avatar
Ignacio Corcuera committed
58 59 60 61
          }


Plugins Code
J. Fernando Sánchez's avatar
J. Fernando Sánchez committed
62
============
63 64 65 66 67 68

The basic methods in a plugin are:

* __init__
* activate: used to load memory-hungry resources
* deactivate: used to free up resources
J. Fernando Sánchez's avatar
J. Fernando Sánchez committed
69
* analyse_entry: called in every user requests. It takes in the parameters supplied by a user and should yield one or more ``Entry`` objects.
70 71 72

Plugins are loaded asynchronously, so don't worry if the activate method takes too long. The plugin will be marked as activated once it is finished executing the method.

J. Fernando Sánchez's avatar
J. Fernando Sánchez committed
73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115

Example plugin
==============

In this section, we will implement a basic sentiment analysis plugin.
To determine the polarity of each entry, the plugin will compare the length of the string to a threshold.
This threshold will be included in the definition file.

The definition file would look like this:

.. code:: yaml

          name: helloworld
          module: helloworld
          version: 0.0
          threshold: 10


Now, in a file named ``helloworld.py``:

.. code:: python

          #!/bin/env python
          #helloworld.py

          from senpy.plugins import SenpyPlugin
          from senpy.models import Sentiment


          class HelloWorld(SenpyPlugin):

              def analyse_entry(entry, params):
                  '''Basically do nothing with each entry'''

                  sentiment = Sentiment()
                  if len(entry.text) < self.threshold:
                      sentiment['marl:hasPolarity'] = 'marl:Positive'
                  else:
                      sentiment['marl:hasPolarity'] = 'marl:Negative'
                  entry.sentiments.append(sentiment)
                  yield entry


116 117
F.A.Q.
======
J. Fernando Sánchez's avatar
J. Fernando Sánchez committed
118 119 120 121 122 123 124 125
Why does the analyse function yield instead of return?
??????????????????????????????????????????????????????

This is so that plugins may add new entries to the response or filter some of them.
For instance, a `context detection` plugin may add a new entry for each context in the original entry.
On the other hand, a conveersion plugin may leave out those entries that do not contain relevant information.


J. Fernando Sánchez's avatar
J. Fernando Sánchez committed
126 127
If I'm using a classifier, where should I train it?
???????????????????????????????????????????????????
128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153

Training a classifier can be time time consuming. To avoid running the training unnecessarily, you can use ShelfMixin to store the classifier. For instance:

.. code:: python

          from senpy.plugins import ShelfMixin, SenpyPlugin

          class MyPlugin(ShelfMixin, SenpyPlugin):
              def train(self):
                  ''' Code to train the classifier
                  '''
                  # Here goes the code
                  # ...
                  return classifier

              def activate(self):
                  if 'classifier' not in self.sh:
                      classifier = self.train()
                      self.sh['classifier'] = classifier
                  self.classifier = self.sh['classifier']
              
              def deactivate(self):
                  self.close()

You can speficy a 'shelf_file' in your .senpy file. By default the ShelfMixin creates a file based on the plugin name and stores it in that plugin's folder.

Ignacio Corcuera's avatar
Ignacio Corcuera committed
154 155 156 157 158 159 160 161
I want to implement my service as a plugin, How i can do it?
????????????????????????????????????????????????????????????

This example ilustrate how to implement the Sentiment140 service as a plugin in senpy

.. code:: python

          class Sentiment140Plugin(SentimentPlugin):
J. Fernando Sánchez's avatar
J. Fernando Sánchez committed
162 163
              def analyse_entry(self, entry, params):
                  text = entry.text
Ignacio Corcuera's avatar
Ignacio Corcuera committed
164 165 166
                  lang = params.get("language", "auto")
                  res = requests.post("http://www.sentiment140.com/api/bulkClassifyJson",
                                      json.dumps({"language": lang,
J. Fernando Sánchez's avatar
J. Fernando Sánchez committed
167
                                                  "data": [{"text": text}]
Ignacio Corcuera's avatar
Ignacio Corcuera committed
168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187
                                                  }
                                                 )
                                      )

                  p = params.get("prefix", None)
                  polarity_value = self.maxPolarityValue*int(res.json()["data"][0]
                                                             ["polarity"]) * 0.25
                  polarity = "marl:Neutral"
                  neutral_value = self.maxPolarityValue / 2.0
                  if polarity_value > neutral_value:
                      polarity = "marl:Positive"
                  elif polarity_value < neutral_value:
                      polarity = "marl:Negative"

                  sentiment = Sentiment(id="Sentiment0",
                                      prefix=p,
                                      marl__hasPolarity=polarity,
                                      marl__polarityValue=polarity_value)
                  sentiment.prov__wasGeneratedBy = self.id
                  entry.sentiments.append(sentiment)
J. Fernando Sánchez's avatar
J. Fernando Sánchez committed
188
                  yield entry
Ignacio Corcuera's avatar
Ignacio Corcuera committed
189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221


Where can I define extra parameters to be introduced in the request to my plugin?
?????????????????????????????????????????????????????????????????????????????????

You can add these parameters in the definition file under the attribute "extra_params" : "{param_name}". The name of the parameter has new attributes-value pairs. The basic attributes are:

* aliases: the different names which can be used in the request to use the parameter.
* required: this option is a boolean and indicates if the parameters is binding in operation plugin.
* options: the different values of the paremeter.
* default: the default value of the parameter, this is useful in case the paremeter is required and you want to have a default value.

.. code:: python

          "extra_params": {
             "language": {
                "aliases": ["language", "l"],
                "required": true,
                "options": ["es","en"],
                "default": "es"
             }
          }

This example shows how to introduce a parameter associated with language.
The extraction of this paremeter is used in the analyse method of the Plugin interface.

.. code:: python

          lang = params.get("language")

Where can I set up variables for using them in my plugin?
?????????????????????????????????????????????????????????

J. Fernando Sánchez's avatar
J. Fernando Sánchez committed
222
You can add these variables in the definition file with the structure of attribute-value pairs.
Ignacio Corcuera's avatar
Ignacio Corcuera committed
223

J. Fernando Sánchez's avatar
J. Fernando Sánchez committed
224
Every field added to the definition file is available to the plugin instance.
Ignacio Corcuera's avatar
Ignacio Corcuera committed
225 226 227 228 229 230 231 232

Can I activate a DEBUG mode for my plugin?
???????????????????????????????????????????

You can activate the DEBUG mode by the command-line tool using the option -d.

.. code:: bash

J. Fernando Sánchez's avatar
J. Fernando Sánchez committed
233 234 235 236 237 238 239 240 241
   senpy -d


Additionally, with the ``--pdb`` option you will be dropped into a pdb post mortem shell if an exception is raised.

.. code:: bash

   senpy --pdb

Ignacio Corcuera's avatar
Ignacio Corcuera committed
242

243 244 245 246
Where can I find more code examples?
????????????????????????????????????

See: `<http://github.com/gsi-upm/senpy-plugins-community>`_.