12. Methodology for evaluating contributions

At this point you should already have a PubPub community, a series of pages with information, invitations sent and a very clear objective of what your users are reviewing. But what happens when the user's input about reviewing your information begins?

The CONL team created a methodology in order to be able to properly use the comments generated in the different Pubs. In turn, we designed another small guide to understand how to take advantage of documents uploaded to the platform. In general, these two processes are simple and highly replicable.

For comments on documents uploaded by the administrator, the methodology consists of 3 phases: identification, analysis and resolution. These phases are accompanied by a database that can capture and organize comments more efficiently (after discerning their relevance to the process).

Identification

There are five possible types of comments that must be defined from the beginning in order to understand what we may face.

Typologies:

Approvals and disapprovals
- This is the most versatile of the five types and can be a precursor or closure for any of the other types.
- The relevance of this comment rests on whether or not it has substance in terms of disapproval (if it is approving, it is not relevant for analysis).
- These types of comments can be left out of the analysis without having to go to the next stage.

Textual suggestions
- Basically, these are comments that propose the change of words, sentences or entire paragraphs.
- They refer to questions of style, semantics, etc.
- They can also be accompanied by approval, disapproval, or even theoretical arguments.

Arguments focused on theory or reliable sources
- These comments are the most complicated since they involve points of view and biases.
- However, they are the most important as they can make great contributions to the review if handled properly.
- The relevance of these arguments lies in their sources, in the validity of their data and theoretical frameworks, and in how much they invalidate or disapprove postulates of the documents that are immovable (example: they contradict the human rights approach or propose a theory to analyze the social part that is already expired) as well as in the breadth of the reference framework used to define them (for example: it is in ODS, work tables, previous plan and public consultation).
- It is very likely that most people will not cite or be explicit about reliable sources of information. You have to be careful with this situation.

Arguments focused on alternative points of view
- These arguments are based on personal or religious beliefs, or imply points of view of doubtful origin.
- It is very easy to identify religious and personal comments, but those that cannot be identified need to be analyzed in detail.

Citizen proposals
- This typology was not contemplated for the exercise, but it had to be included since many citizens used the platform to express ideas to improve the State. Although this was not the function of the exercise, the comments were taken into account since they could be used for other exercises or as inputs to create projects.

Analysis

Once each of the possible types of comments is identified, they must be analyzed. For this, it is enough to have a small model that is based on the form and background of the comment. It is at this time that comments containing the five typologies can be identified, since some types may be focused on form, but have a more important background (substance).

It is important to say that all comments have form and substance, but due to the type of exercise that is carried out, the substance may not be identifiable. For this reason, in addition to the fact that a comment can be identified by one of the five previous typologies, it will also fall in one way or another in one of the two proposed categories: form and substance.

Form: Comments without concise arguments, or data, from unidentifiable frames of reference. They are mainly textual suggestions and / or approvals.

Substance: structured comments that seek to analyze the text from other perspectives. They propose alternative frames of reference or give arguments that justify disapproval, or reinforce the text.

Form

Substance

Comments that are only considered formally are those that have no context or express no more than unreasonable suggestions or approvals / disapprovals.
There is the possibility that this type of comment starts with one typology and ends with another. It is necessary to take this into account, since where there is a disapproval there may be a hidden background that cannot always be identified.
It is very important to be clear when comments without background can be analyzed.
Any disapproving comment will have to be reconsidered only in the event that:
- More than 5 people express disapproval
- OR the team considers it relevant to reflect on that disapproval or suggestion.

In the case of comments that provide textual suggestions and do not have identifiable arguments, they will have to be analyzed by the team to decide whether or not the wording change is feasible.

Note: the team cannot focus on this type of analysis.
The type of textual suggestions that come with a background will be prioritized.

The case where a suggestion is always nullified is in the case that the textual suggestions go against inclusive language or any attempt to make any problem or group invisible, also against some higher principles like human rights.

All comments have a background, but due to the virtual mechanism and the impossibility of generating discussions between the team and the participants, only comments with arguments can be analyzed from the substance.
Any comment that only says yes or no; a change in the text; or a proposal of data or theories without validity, should be considered as a contribution with a possible little validity or importance.
If the arguments presented are outdated and correspond to ideas that no longer have theoretical validity or scientific support, they should be automatically discarded.

This process is only feasible when the source or author is denoted and there is a way to verify the information.

If the arguments presented are morally charged or refer to religious views and beliefs, they should be automatically discarded.
If the arguments presented have a discriminatory connotation, they should be automatically discarded.
If the arguments presented express that the data of the information is not correct, but the participant does not express the source that says otherwise, it is the responsibility of team to review it.
If the arguments go against the theories and support of the information, but there is information on the part of the user who commented that it could be used to debate what was used in the construction of this theoretical framework, it is the responsibility of team to review it.

Summary: Automatically Disapproved

The automatically disapproved Comments are:

Approvals (I agree).
Disapprovals with less than 5 comments (this is entirely discretionary and each practice management team can change this rule).
Textual modifications without background
If the suggestion is quite precise and a change is necessary, it is taken as a comment
If it is just something that takes away the meaning or is grammatically incorrect, it is not taken into account
The recommendation in this sense of language is that, as far as possible, all these comments are reviewed by someone who is an expert in this area and the theoretical part that needs more analysis is left to the experts in each area.
Textual modifications that go against inclusive language or any attempt to make any problem or group invisible, also against human rights..
Religious arguments are automatically disapproved (obviously if the consultation has some characteristic about some religious belief, it will be totally valid).
That they violate the rules of coexistence (more about this in the section of coexistence rules, terms and conditions and privacy notice.)

Comments subject to debate:

Arguments based on personal experiences can serve as testimonials to see if a topic was missing. However, it must be understood that this topic is very likely already in the document or is something outside the scope and responsibilities of the institution that prepares the consultation, in which case it is disapproved.
Very well crafted arguments without sources. They are likely to have no sources due to the nature of the process. However, it is necessary to analyze them to see what they imply.
Everything that was a proposal or project idea was reflected in the database. It is always important to see how much they are relevant to the exercise. For example, there were certain ideas and proposals that were within the scope of the federal executive or the municipalities. This helps decision makers to understand what problems people face, but at the same time, it shows us that not everything can be used due to the responsabilities of the institutions.

Resolution and capture of comments

Once the comments have been identified and analyzed, a verdict is given as a conclusion of what will happen with that comment. This verdict should be at the same time as the comment capture is taking place. This is only for a matter of time, since capturing all the comments in a database and then analyzing them is a huge time consuming and in this type of process, time is crucial.

Since PubPub does not allow automatic downloading of comments, each approved comment will be transferred to a database. This will help us to have analytics of what types and categories of comments are put, as well as to have control between chapters.

The comment is placed in the database with its respective typology;
And the username.
Done… the proposed change occurs in the review process.
Comments from the same discussion can be stacked.

Note: we recommend using the "blog" comment display functionality found under each Pub. You can also see the comments in each of the comment bubbles on the right side of the Pub, but this becomes very complicated when you have many contributions.

An example of a database can be consulted at the following link: https://participa.conl.mx/pub/5c33k03d/release/1

In this example of a database you can see two important things:

There is a sheet that helps the comment capturer to understand what the typologies are. This tab is for guidance only and has no other function.
The other sheets represent each of the different Pubs to be analyzed.
- Example 1 is the most traditional of the Pubs analysis capabilities.
  - This type of sheet is recommended when there are few comments.
- Example 2 is technically the same as Example 2, but a column named “code” was added that uses a semaphore to identify certain types of comments.
  - This was decided during the analysis and especially for this Pub as this example had an excessive amount of comments.
  - The traffic light denotes the priority to integrate and use the contributions as inputs.
    - Red denotes that the comment is not relevant enough to be integrated at this time, but will be considered later.
    - Yellow denotes comments that are important, but due to lack of time cannot be integrated at this time.
    - Green denotes inputs that are important and that are so relevant to the process that they should be used as input.

Let us remember that this database example is only a possibility and that each administrator of the query process can create new models and processes. This example worked for the Consejo Nuevo León and should not be understood as a rule for other types of processes. You can use colors too!