Navegação – Mapa do site

InícioNúmeros44Artigos TemáticosThe Strategic Approach to Quality...

Artigos Temáticos

The Strategic Approach to Quality in Journalism: Innovation, Technology and Applied Research

A Abordagem Estratégica da Qualidade em Jornalismo: Inovação, Tecnologia e Pesquisa Aplicada
Josenildo Luiz Guerra
Tradução de Anabela Delgado
p. 1-18
Este artigo é uma tradução do:
A Abordagem Estratégica da Qualidade em Jornalismo: Inovação, Tecnologia e Pesquisa Aplicada [pt]

Resumos

O presente artigo aborda o tema da qualidade em jornalismo a partir de uma perspectiva estratégica (Paladini, 2005), segundo a qual a qualidade é um esforço organizacional que considera condições e demandas do cenário externo articuladas com as ações internas, desde o planejamento até a entrega do produto final. Para implementar essa perspectiva, será necessário pensar a qualidade em jornalismo para além da concepção majoritariamente focada na avaliação do produto, para a concepção baseada no desenvolvimento de sistemas de gestão da qualidade (SGQ). O SGQ é um conjunto de ações integradas, que alinha a avaliação a diretrizes da gestão editorial, processos de produção e produtos. A implementação dessa proposta requer o investimento em inovação, em dois movimentos complementares, para desenvolver: (a) um SGQ para organizações jornalísticas, que contenha métodos de avaliação com métricas para mensuração de resultados e modelos de gestão editorial, processos e produtos que incorporem os métodos de avaliação e suas métricas; e (b) sistemas informatizados capazes de permitirem a operacionalização do SGQ, através dos recursos tecnológicos disponíveis nos termos do jornalismo estruturado, que explora as potencialidades inscritas no jornalismo digital em base de dados. A abordagem estratégica da qualidade em jornalismo não se viabiliza com o know-how disponível hoje no setor. O investimento em pesquisa aplicada e no desenvolvimento experimental em jornalismo será fundamental para gerar inovação editorial e tecnológica, a fim de operar a transformação requerida para a construção de uma nova inteligência jornalística, capaz de elevar a credibilidade, com dados de qualidade editorial monitoráveis e demonstráveis, e garantir a sustentabilidade do setor, no longo prazo.

Topo da página

Notas da redacção

DOI: 10.17231/comsoc.44(2023).4736
Submetido: 11/04/2023 - Aceite: 13/11/2023

Texto integral

1. Introduction

  • 1 The term “journalistic institution” refers to the collective knowledge and interested parties (prof (...)

1The technological transformations affecting the global media sector have profoundly impacted journalism. To a greater or lesser extent, those changes threaten the business sustainability and the credibility of the journalistic institution1 and its organisations as reliable sources of information (Anderson et al., 2012). Specifically regarding quality, two examples underscore the urgency for journalistic institutions to respond to this challenging landscape that jeopardises their credibility. Recent surveys conducted by the Reuters Institute have shown a consistent decline from 2015 to 2022 in the percentage of people who trust most of the news they consume across several countries surveyed (Newman et al., 2022). Equally recent, quality assessment studies (Guerra, 2020; Rothberg & Garrido, 2019; Träsel et al., 2019) indicate an incipient adoption of good editorial practices in requirements directly linked to credibility. This sets a critical scenario for the business model, currently associated with a low perception of quality among audiences and modest outcomes in the assessment processes implemented.

2Specifically in the case of online media, the potential of which is the subject of this work, Morales-Vargas et al. (2022) emphasise the pressing need for rigorous and reliable journalism amid the current widespread dissemination of fake news. However, their extensive review of quality indicators in this domain revealed a notable gap, specifically in assessing reliability indicators. Considering the convergence landscape (technological, entrepreneurial, professional and communicative) described by Salaverría (2003), this study seeks to associate editorial quality efforts, in which trust requirements stand out, with resources specific to online media, such as those associated with digital journalism in databases (DJDB), which underpins the entire journalistic production environment. The goal is to merge both sets of indicators, those based on media characteristics (Palacios, 2011) and those focused on content reliability, to create more comprehensive assessment tools.

3This study is based on the premise that quality is a distinctive element (Fengler et al., 2014; Meyer, 2004/2007), not only among news organisations that seriously embrace the subject compared to others that ignore it but also among new competitors, such as social media. An editorial quality policy consists of demonstrating virtues worthy of earning public trust.

4However, implementing the quality agenda is only part of the effort, which should be coupled with a proposed set of solutions designed to elevate journalism to a higher standard of professional proficiency. In order to develop this proposition, this article adopts a methodological structure grounded in applied research and experimental devel-opment, the basic foundations of which will be presented before revisiting the underlying theme of quality and its implications.

5The proposed movement entails rethinking quality management to harmonise innovation in editorial management, production processes and products with technological innovation. Innovation should be understood as “a new or improved product or process (or a combination thereof) that differs significantly from the unit’s previous products or processes and that has been made available to potential users (product) or brought into use by the unit (process)” (Organisation for Economic Co-operation and Development, 2018, p. 32). Embracing innovation becomes imperative to meet more rigorous quality standards.

6This approach requires universities and organisations to invest in applied research, a type of research directed primarily “towards a specific, practical aim” (Organisation for Economic Co-operation and Development, 2015, p. 51). Applied research outcomes are intended primarily to be valid for possible applications “to products, operations, methods or systems” (p. 51), which become the object of experimental development,

systematic work, drawing on knowledge gained from research and practical experience and producing additional knowledge, which is directed to producing new products or processes or to improving existing products or processes. (Organisation for Economic Co-Operation and Development, 2015, p. 51)

  • 2 To explore further insights and references on applied research in journalism, consider works by Med (...)

7Based on this understanding, the history of research within the scope of the research programme in quality, innovation and technology applied to journalism has systematised an applied research method in journalism22. It encompasses various theoretical and methodological elements focused on understanding how journalism is done in order to intervene in it with a scientific approach. It aligns theoretical perspectives with professional practices, developing the foundations and conceptual developments to address the normative, technical, and procedural aspects that shape the activity.

8The process begins with an initial understanding of what journalism is, from which problems are formulated, and the applied effort of conceptual and methodological systematisation seeks to offer an innovative solution. Initially conceptual, this solution might evolve into an experimental development phase, transforming into a framework meant for practical implementation, subject to testing, adjustments, and revisions to validate its efficacy in solving the problem it was designed to solve.

9Applied research in journalism is guided by two key principles: the purpose principle and the complementarity principle. The purpose principle organises the responsibilities the journalistic institution proposes or is asked to accomplish in society through agreements and conventions established among interested parties, incorporating quality requirements pursued by journalistic organisations. The complementarity principle determines the acknowledgement of various axes that, although addressing distinct and specific issues, converge in a particular configuration to impart meaning and effectiveness to the purpose principle. The integration among these axes demonstrates coherence between the formulation of the problem, the proposed solution and the various types of knowledge used to implement it.

10The six axes of the complementarity principle are defined in Table 1 and broken down according to their application in this article.

Table 1. Axis of the complementarity principle related to the problem and solutions developed in this proposal

Axis of the complementarity principle

Application

Theory

Theoretical and methodological foundations outlined in the literature review, in the proposal for a strategic approach to quality in journalism, the definition of a quality management system (QMS) and the connections suggested between editorial fundamentals and technological resources, such as the concepts of “digital journalism in databases” and “structured journalism”

Ethics

The primary and secondary requirements proposed (which we will discuss later) derive from fundamental ethical values inherent in democratic societies aligned with foundational concepts shaping both the problem and the proposed solutions

Technique

The technical procedures have yet to be developed as they are contingent upon the editorial and technological decisions stemming from the proposed solutions. Advancement in this axis hinges on consolidating the conceptual understanding of the strategic approach to quality in journalism and the QMS

Processes

The process is envisioned as a production domain defined by parameters set by editorial management guidelines and standardised requirements outlined by the QMS. Moreover, this process is connected to technological advancements in software development to manage and implement its actions and measurements concerning quality

Technology

Technology is an essential resource in editorial management, whose functionalities can facilitate both the quality-oriented management of processes and the measurement of indicators demonstrating the organisation’s attained quality levels

Sustainability

Sustainability is not directly developed, as there are no analyses concerning the investment and potential return of the QMS proposal and its developments. From the point of view of the journalistic business model, quality management can elevate both the product quality and the interested parties’ perception of quality, which contributes to bolstering credibility over the medium to long term, thereby offering a competitive edge in the market

11This work addresses the prevalent problem in contemporary journalism, wherein the existing model lacks the mechanisms within its processes and products to plan and demonstrate its effectiveness in delivering credible content. Mere declarations of commitment to quality are insufficient unless accompanied by measurable outcomes demonstrating its excellence. If this situation prevails, there will be no reaction to the loss of trust and no proactive action to strengthen it.

12The proposed solution unfolds in two steps:

Conceptual design of a quality management system (QMS) aimed at planning, implementing and assessing quality in journalistic organisations. It aims to simultaneously and complementarily steer production towards quality, consistently assess outcomes to correct flaws and seek continuous improvement in journalistic production. Beyond addressing failures, the overarching goal of the “system” is to attain editorial quality and showcase its positive outcomes in order to give society and other interested parties reasons to trust its products;

The second movement outlines a roadmap for applied research and experimental development essential for implementing the proposed conceptual QMS, seamlessly combining editorial and technological solutions.

13Given the effort to articulate quality with innovation, technology and applied research, the text endeavours to make basic connections and concepts to support this movement. As such, it intentionally omits certain details due to space constraints. This proposal will unfold across three sections. The first, a brief literature review, explores

14(a) points of convergence between quality requirements across different works and (b) the alignment of approaches to the different components of a QMS. The direction in the previous topic will underpin the second topic, the strategic approach to quality in journalism (SAQJ), which shapes the contours of the proposed solution, delineating a conceptual QMS applied to journalistic organisations. The third topic argues that the QMS concept requires applied research and experimental development to generate the editorial and technological innovation needed to implement it. These solutions require innovative editorial and technological competencies, which can form the groundwork for a new journalistic intelligence.

2. Grounds for a Strategic Perspective on Quality in Journalism

15The acknowledgement that studies on journalism quality encompass diverse approaches, assessment parameters, and evaluation methods is evident (Santos & Guazina, 2020). Nevertheless, amidst this diversity, there is a recognition that various theories and methods face shared challenges. Considering the urgency of this issue for the future of journalism, there is a call to identify convergence points that will allow the academic community, organisations and professionals to advance with enhanced systematisation and deeper insights into the subject matter.

16Within its original realm, administration, quality assessment seeks to measure the levels of suitability of products to the organisation’s internal standards and consumer expectations, encompassing their perceived quality (Berry & Parasuraman, 1991, as cited in Slack et al., 1997/2007; Chiavenato, 2007). This endeavour includes various approaches based on intrinsic (transcendental) excellence, the product, the user, production or value (Garvin, 1987, as cited in Carvalho, 2005). However, a perception of quality as a strategic investment for organisations has overtaken specific approaches to quality.

17The strategic approach to quality must consider “broad scenarios that extend beyond the organisation itself”, encompassing the entire “external area in which it operates” (Paladini, 2005, p. 26). This includes suppliers, customers, markets, and competitors, among others, without excluding the tactical and operational management levels. Pursuing quality is a guiding principle in organisational action, harmonising internal and external factors to ensure public recognition and the organisation’s long-term sustainability.

18Embracing quality as an operating principle within an organisation requires a management model that translates this principle into reality. For instance, the quality standards of the International Organization for Standardization, whose requirements “are generic and are intended to be applicable to any organisation, regardless of its type or size, or the products and services it provides” (Associação Brasileira de Normas Técnicas, 2015b, p. 1). These standards propose an approach to conceiving and managing quality that is considered the most effective and can be applied to journalism (Swanepoel, 2012; Torre & Téramo, 2007).

19The challenge is not merely adopting these references but adapting them to a specific model for journalistic organisations, especially due to a lack of knowledge within journalism regarding management, processes, and assessment tools ingrained in the professional and organisational culture. Overcoming this gap is crucial for strategic quality management.

20Fundamental principles outlined by the Brazilian Association of Technical Standards (Associação Brasileira de Normas Técnicas, 2015b, p. 1) will serve as guiding pillars in addressing this challenge and constructing the proposed model. The goal is to demonstrate, in two movements, significant indicators of alignment pointing to the viability of this endeavour: firstly, delineating quality assessment requirements in journalism; secondly, assessing the suitability of the various studies, to varying degrees, with pertinent aspects underpinning the SAQJ proposal advocated here.

2.1 Quality Requirements in Journalism: Points of Convergence

21Quality refers to the extent to which a product meets requirements — stipulated needs and/or expectations, explicitly stated or implied which “can be generated by different interested parties or by the organisation itself” (Associação Brasileira de Normas Técnicas, 2015a, p. 21). Requirements are fundamental because they establish benchmarks against which quality is assessed. Within studies focusing on quality in journalism, there is a diversity that needs to be equated in order to build basic consensus.

22This study draws on Santos and Guazina (2020) and Chistofoletti (2010) to illustrate the convergence between different propositions. Santos and Guazina (2020) conducted a survey encompassing North American, European, and Latin American studies, and their analysis mapped the diversity of perspectives on the topic. Chistofoletti (2010) compiled a list based on interviews with 22 editorial managers from Brazilian companies. Table 2 outlines the requirements they explored, categorised here into affinity groups based on their initial research.

Table 2. Overview of journalistic quality requirements

Affinity groups

Requirements

Number

Group 1. Emphasis on independence

Acting without government interference; independent; independence; non-partisanship; economic independence; editorial independence

6

Group 2. Emphasis on impartiality and balance

Impartiality (three references); diversity; equity; proportionality; balanced and fair information, reporting alternative perspectives and interpretations in a nonsensationalised and impartial way; plurality (two references); evidence of integrity (fairness, balance or independence)

10

Group 3. Emphasis on demands of democratic societies

Participatory democracy; contributing to an informed society or media truth; providing the basis for democratic decision-making processes; public interest; social concern; alertness to risks; social relevance; comprehensive offer of relevant news and general information about events in society and the world around it

8

Group 4. Emphasis on truth

Accuracy (three references); investigation (two references); depth of investigation; the best version of the truth; objectivity (two references); rigour of reporting; objective information, in the sense of being factual in form, accurate, honest, sufficiently complete and true to reality; reliable, in the sense of being verifiable in fact of opinion; clean, noiseless reporting; zero error rate; good reporting; information and not opinion; reliability; correctness

18

Group 5. Emphasis on relevance

Relevance; social relevance; informative zeal or relevance; prioritisation and the relationship between contents; geographical and emotional proximity of the information; novelty; originality of the subject; meeting the public’s daily information needs

8

Group 6. Emphasis on product

Content of stories; quality of writing; clarity of story; clarity of style; pleasing presentation; quality of presentation; simplicity

7

Group 7. Emphasis on

staff profile

Diverse staff; courageous; expand educated audience; desire to influence opinion leaders; specialised staff

5

Group 8. Diversity and connection to purpose

Protection from propaganda and irrational appeals; desire to help; absence of sensationalism; service to the reader; responsibility; social responsibility; commitment to the community; context; loyalty; ethics; criticism; alignment with the public; usefulness to the reader

13

Group 9. Miscellaneous, connection with product characteristics

Quality of analysis; attention to context; good stories and interesting characters; elements that allow the facts to be interpreted; journalistic scoop; interpretation of reality and going beyond the facts; speed; novelty; differentiated approaches; breadth; credibility; efficiency; selection of existing and reliable facts and sources; the sources; the choice of agenda

15

Note. Based on a literature review by Santos and Guazina (2020) and Christofoletti (2010)

23The survey indicates dispersion due to terminological variation among requirements, yet it also highlights potential convergence when grouped by affinity.

2.2. Points of Convergence Among Journalism Studies on Quality and the Quality Management System

24Attaining quality requires a QMS, the set of activities through which an organisation identifies its objectives and determines the processes and resources needed to attain the desired outcomes. Journalism studies considered individually explore, to some extent, aspects relevant to what a QMS requires. However, they do not constitute a system because they do not encompass a “set of interrelated or interacting elements” (Associação Brasileira de Normas Técnicas, 2015a, p. 19).

25Following, we will outline the seven components of a QMS according to the Brazilian Association of Technical Standards (Associação Brasileira de Normas Técnicas, 2015b) standard, linking them to various types of quality research produced in journalism studies. This comparison aims to illustrate how these research types, while not directly aligned with the standard, resonate with the concepts it encompasses.

Context of the organisation: definition of external and internal issues relevant to the organisation’s strategic direction, which impact its ability to achieve outcomes considering the expectations and needs of interested parties. Studies on quality in journalism delve into the origin of requirements segmented by interested parties, falling into two groups. One group focuses on identifying actors directly involved in the production, such as journalists, organisations, academia, the audience and their respective demands for quality (Lacy & Rosenstiel, 2015; Vehkoo, 2009/2010). This group includes pressures from political and economic actors, for example, as highlighted by Odriozola-Chéné et al. (2019), based on interviews with Latin American journalists, who express concerns about their editorial autonomy being compromised by interference in their routines, impacting the quality of their work. The other group addresses macro-interested parties, including the market, legal system, media system and the public, along with their demands concerning journalistic products (Bertrand, 1999/2002; Fengler et al., 2014; McQuail, 2003, 2013; Mompart & Sampio, 2013; Pinto & Marinho, 2003). This approach involves mapping interested parties capable of proposing quality requirements to guide the work of organisations. However, the requirements stemming from these segments are often depicted as conflicting or competitive. In this sense, assessing quality from one perspective could compromise it from another. As a rule, each perspective has its unique approach, which may diverge or converge with other perspectives. The SAQJ aims to make these requirements converge as much as possible.

Leadership: establishing purposes, commitments and responsibilities, expressed in the quality policy, focusing on the customer and interested parties. Torre and Téramo (2007) state that adopting a QMS should be a decision by the top management of news organisations. Initiatives such as The Trust Project (2023), Journalism Trust Initiative (Reporters Without Borders, 2021) and News Guard (2023) have been developed to assess organisations’ commitment to practices and values that increase their credibility. All these initiatives’ indicators rely on the leadership’s decision within an organisation to adopt them. In other words, by analysing the availability of the required tools, these studies assess not just the mere existence of these instruments but the commitment of organisational leaders toward their adoption.

Planning: includes preceding elements to ensure that the QMS achieves its outcomes, prevents and avoids adverse effects and fosters improvement. This group encompasses studies suggesting the evaluation of editorial planning documents, such as writing and style manuals, editorial projects, and charters of principles, among others, which provide operational guidelines for journalistic organisations. Examples include works by Rothberg and Garrido (2019), Guerra (2010), Christofoletti (2010) and Swanepoel (2012).

Support: ensuring the resources, people, infrastructure, environment, monitoring, measurement, and more for implementing the QMS. This item is explored in works such as that proposed by the labour index (Pablo & Mateos, 2004, as cited in Mompart & Sampio, 2013), which assesses indicators concerning working conditions provided to professionals or when assessing, for example, the training processes offered by organisations to their staff. The journalists interviewed by Odriozola-Chéné et al. (2019) highlighted that the lack of material resources, leading to insufficient time, tools and even compromised salaries, also impacts the attainment of minimum quality standards.

Operation: operational planning and control for product provision, requirement determination, customer communication, critical analysis, product design and development, and related aspects. It involves works directly or indirectly targeting the production process, from the selection of information to its public dissemination. Gordillo et al. (2013), for example, explain that the Journalistic Added Value method distinguishes processes from journalistic products and establishes that it is possible to analyse the process based on the published product (Pellegrini, 2011, as cited in Gordillo et al., 2013). The research that best focuses on processes is newsmaking studies. However, they do not aim to assess quality, as they use methods that involve becoming deeply embedded within the organisation to understand the production process internally.

Performance evaluation: encompasses monitoring, measurement, methods, analysis and evaluation, including the treatment of outcomes. It is the most developed item in work on quality in journalism, and it covers three primary areas: requirements (as previously discussed in Section 2.1), the object, and the evaluation method, all detailed below. Regarding the object, evaluation objects can be distinguished between product and package. The former denotes the specific information unit (a news article), and the latter encapsulates a set of information units (a programme or edition). These units exhibit content characteristics (the information) and characteristics related to drafting and style (the message). There are various classifications, including those by van Cuilemberg (2000, as cited in Lacy & Rosenstiel, 2015), Rosengren et al. (1996) and the “journalistic index” (Pablos & Mateos, 2004, as cited in Mompart & Sampio, 2013). Regarding the methods, Lacy and Rosenstiel (2015) systematised four methods: (a) product analysis, applied to assess political balance, variety of coverage, and sources, among others; (b) audience preferences, conducted through surveys; (c) expert judgement, involving surveys with experts, editors, professionals and professional awards; and (d) indirect indicators, such as financial outcomes, total news/team count; news per agency service; among others.

Improvement: involves determining opportunities for improvement and implementing the actions needed to meet client requirements, encompassing corrective action and continuous improvement. Contributions come in two forms within this context: direct and indirect. Direct, when assessments outline the desired requirements and encourage their adoption by those assessed, often offering some form of recognition or reward. For example, The Trust Project (2023), Journalism Trust Initiative (Reporters Without Borders, 2021) and News Guard (2023) award quality certifications to organisations meeting their indicators. Indirectly, by establishing indicators, every evaluation suggests to those being evaluated what constitutes a commendable journalistic product according to that evaluation method. However, unlike direct contributions, indirect ones lack a clear procedure for indicating these requirements, making it challenging for evaluated organisations to incorporate them effectively.

26In summary, the brief analysis of the requirements shows how they converge towards potential consensus parameters despite the diversity of terms. Furthermore, the relationship between studies on quality in journalism and the components of a QMS reveals varying degrees of emphasis on complementary aspects when viewed from the strategic approach perspective.

3. Towards a Strategic Approach to Quality in Journalism

27In strategic quality assessment, dealing exclusively with quality assessment is not enough. It is necessary to consider the QMS, the set of procedures to plan and implement quality in a given organisation. The assessment is just one facet among a spectrum of actions that must be strategically orchestrated and executed cohesively. Hence the notion of “system”.

28This topic aims to distil the guidelines outlined in the Brazilian Association of Technical Standards (Associação Brasileira de Normas Técnicas, 2015a) standard for QMS, offering a roadmap to address the specific demands within journalism.

29Figure 1, mapping the external and internal contexts of the organisation, illustrates the intricate involvement of diverse QMS components within journalistic activities (Guerra, 2020).

Figure 1. Concise diagram outlining the interactions, interested parties, and organisational dynamics intended to form a quality management system applied to journalistic organisations

Figure 1. Concise diagram outlining the interactions, interested parties, and organisational dynamics intended to form a quality management system applied to journalistic organisations

Note. Created by the author. An earlier, slightly different version is available in Guerra (2020)

Context of the organisation: Figure 1 expresses the aspects that shape the organisation’s context. It illustrates interested parties expressing their needs and expectations, with varying degrees of influence determined by economic and political prowess. Market dynamics typically wield significant influence due to their direct impact on an organisation’s financial viability. In societies with media regulation systems, interested parties from the public and legal framework axes have strengths that can counterbalance the strength of the market axis compared to societies that do not have them. Organisations draw from this interaction the editorial requirements they deem viable to make up the product they want to offer their audience, the basis for their financial sustainability, while also meeting the demands of other interested parties, in whole or in part.

Leadership: leaders assume the responsibility of steering the negotiation process internally, using the filters they deem relevant, such as values, resources, and infrastructure, and applying them to define the requirements that should shape their journalistic product. Leaders also oversee QMS implementation and provide the operational means to consistently assess the alignment between production and the defined requirements.

Planning: organisations need to convert the requirements into products, processes and technical specifications, which should provide the references for operating the QMS, both to assess and to guide production. In Figure 1, planning is expressed by the “editorial management” field, encapsulating the requirements stemming from the negotiation process with interested parties. Manuals, editorial projects and similar documents are examples of planning already available in organisations. However, in many cases, they are insufficient because they do not incorporate instruments for evaluating the performance of the requirements they claim to follow.

Support: necessary to guarantee the means to implement planning actions, such as personnel, training, facilities and equipment, among others.

Operation: operation stands as the core activity within the organisation. In Figure 1 it is represented by the processes integral to executing editorial planning. This includes operational planning, editorial decision-making, executing production stages with their respective deliveries, and employing content production techniques (research, drafting and editing, for example), among other fundamental elements in journalistic production.

Performance evaluation: becomes integral in ensuring that the requirements outlined in editorial planning are upheld throughout the production cycle, ensuring the quality of the final output. It involves consistent evaluation mechanisms at each stage of production. For instance, from the outset, through the sources suggested, the level of diversity within a news story can be gauged, allowing for corrective interventions to balance any biases. In addition to the process, the organisation must regularly monitor whether the requirements established in the planning align with stakeholder demands and their perception of quality. A set of evaluation points becomes essential for the QMS to monitor quality, using pre-established indicators and methods, from the product’s conception through its production to the degree of satisfaction of those who receive it.

Improvement: the evaluation identifies diverse flaws, necessitating interventions to improve the product. Some can be remedied with the existing know-how, while others require innovative solutions. Implementing a QMS requires innovation in management, processes and products, aligning common requirements among interested parties and convergent methods of execution and evaluation.

30The design of a QMS applied to journalism requires clarity on the requirements to be employed, which should guide the entire editorial conception of the product, its production processes and its evaluation methods. Regarding the requirements, (a) although there is a diversity of forms of expression, as illustrated in Table 2, there are identifiable convergences; (b) they must express, to some extent, shared demands and expectations among different interested parties. Otherwise, their quality will not resonate with those whose requirements have not been partially addressed.

31The following is a minimum set of requirements reasonably established both in the theoretical and professional field of journalism and aligned with the normative framework of democratic societies. They represent voluntary, assigned, or contracted responsibilities (McQuail, 2003), which bind the relationship between journalistic organisations and their various interested parties. Their minimal nature indicates a fundamental core that garners maximum consensus, from which it is possible to derive, in an aligned manner, complementary requirements to deal with the complexity of the journalistic production process.

32The requirements are divided into primary and secondary. The primary requirements stem from the deliverables directly linked to journalism’s most basic responsibility, as expressed by Kovach and Rosenstiel (2001): “to provide citizens with the information they need to be free and self-governing” (p. 17). In this sense, three essential requirements (truth, recentness and relevance) are crucial for journalistic information to connect individuals with facts, the present moment, and important issues for decision-making3.

33The secondary requirements stem from journalism’s integration into a particular type of society, in this case, democratic societies. Despite the different democratic cultures and modes of institutionalisation that may shape specific journalistic demands (Anderson, 2014; Cristian, 2009; Strömbäck, 2005), a core of fundamental values can be used as a common reference. It is important to note that democratic societies acknowledge that journalism requires classic prerogatives that provide suitable conditions to fulfil its responsibilities and deliver the expected requirements (Table 3).

Table 3. Set of prerogatives and primary and secondary requirements in democratic societies

Prerogatives

Requirements

Primary

Secondary

Freedom of expression

(information, opinion and criticism) Freedom of the press or journalistic

information

Right to information

Safety

Truth (Group 4)

Relevance (Group 5)

Recentness

Autonomy/independence: within democratic and professional parameters (Group 1)

Public interest: prioritising the public over the private

(Group 3)

Respect for people’s dignity: imposing limits on prerogative use (Group 3)

Plurality and contradiction: addressing legitimate conflict situations (mediation; Group 2)

34The primary and secondary requirements represent convergence points for developing new and essential requirements for the information (the content) and the message (the format). The controlled expansion of requirements ensures coherence and alignment, integrating them into a QMS to direct the entire journalistic production flow, as depicted in Figure 2. This process should establish editorial guidelines and specifications for processes and products aligned with the performance assessment parameters. Each organisation will incorporate these benchmarks to shape its unique editorial approach (for an example of how management guidelines unfold in journalistic production and quality assessment processes, refer to Swanepoel, 2012).

Figure 2. Alignment of requirements to integrate the production process

Figure 2. Alignment of requirements to integrate the production process

35Once these guidelines have been outlined, specifying the structure of a QMS applied to journalism and the minimum requirements that should guide it, the next step involves analysing the challenges for its implementation in journalistic organisations. This requires a coordinated effort between editorial conception, ensuring parameter alignment (Figure 2), and technology to guarantee its implementation (Figure 3).

Figure 3. Structured requirements for a quality management system generate parameters for a computerised editorial production management system focused on quality

Figure 3. Structured requirements for a quality management system generate parameters for a computerised editorial production management system focused on quality

4. Strategic Approach to Quality in Journalism, Applied Research and Innovation in Journalism

36As has already been pointed out, there is no available know-how to implement a QMS along the lines outlined here at the current stage of the journalism industry. No clearly defined parameters, metrics or technical and technological tools can integrate the current production and evaluation processes. In other words, news organisations produce content without generating quality indicators that could certify compliance with declared requirements to themselves and all their interested parties.

37In the previous section, an effort was made to outline (a) a clear set of requirements open for validation by interested parties and (b) a QMS capable of systematising the requirements from their origins through the internal processes of definition, planning, execution, evaluation and improvement.

38The challenge for journalism, as an academic field and an industrial sector, lies in developing more advanced know-how than is currently available to enable accurate editorial planning rooted in requirements endorsed by relevant interested parties, the establishment of production processes guided by verifiable and measurable parameters and evaluation processes anchored in the same parameters.

39As highlighted earlier, this can only be achieved through applied research and experimental development. Assuming that the primary and secondary requirements presented briefly are acknowledged by the interested parties with some degree of consensus, applied research in journalism has three main axes of practical problems to tackle:

Producing diagnoses: (a) which evaluation methods are capable — internally and externally — of gauging and measuring satisfaction levels regarding editorial management, processes, and journalistic products?; (b) which problems identified by these methods need to be addressed?

Development of editorial solutions: (a) develop appropriate evaluation methods to measure requirements in editorial management, processes, products and with interested parties; (b) seek innovative solutions to address the problems identified and devise management tools, implementing processes and new products in which the presence of requirements is measurable and evident;

Development of technological solutions: (a) develop software capable of implementing editorial management, production processes and product presentation, focusing on quality and equipped with performance metrics available across the workflow, from planning to product delivery.

40Effectively addressing the three axes of problems requires a cohesive approach. The previous topics made partial progress on the first two. This one emphasises the third axis, crucial for seamlessly integrating the first two into a news organisation. This will require overcoming “the lack of an elaborate ‘metadata culture’ in information management within editorial workflows” (Pellegrini, 2012, as cited in Lima Junior & Oliveira, 2017), among other challenges, such as the lack of information structuring in the planning, management and implementation of journalistic products.

41To this end, DJDB holds significant untapped potential that “emerges due to the advantages provided by DB [databases] in constructing and managing journalistic websites and structuring and presenting content” (Barbosa, 2008, p. 91). The author points out 18 functions the DJDB can implement, focusing on internal product management, investigative processes, contextualisation, information structuring, news drafting, data retrieval, and content presentation. Such possibilities enable the development of publishing systems that incorporate reporting, message drafting, and content editing subsystems (Schwingel, 2012). Leveraging these technological resources can significantly enhance the quality of journalism.

42Creating a QMS for journalism, according to the specifications outlined here, will require rethinking the journalistic production process to leverage the available DJDB functionalities. It will require a connection with so-called “structured journalism”,

a novel digital journalism trend that contemplates reuse, data accumulation, revision, and comparing current and archived information. It is defined as a system in which database-like storage and access to information is combined with the potential for exploitation and dissemination of digital journalism. (Freixa et al., 2017, p. 1076)

43The existing technology for developing computerised systems is necessary but insufficient because it depends on journalism’s conceptual and operational systematisation, which provides the parameters. This underscores the need for journalism to undergo a reconfiguration based on a well-defined requirements matrix and for the implementation and monitoring to be structured through a QMS. Without this structured framework, any attempt to develop software capable of implementing this proposal would likely fall short in effectiveness.

44Figure 3 illustrates this principle. The definition of primary and secondary requirements should guide the flow of journalistic production through a QMS, which structures the requirements, the processes for obtaining them and the monitoring metrics that will define the parameters of a quality-orientated information system, whose technological viability is inscribed in the possibilities of structured journalism and DJDB.

45Qualijor (Guerra, 2016), a software designed to streamline the flow of quality-orientated journalistic production, is an example of such an endeavour. Figure 4 illustrates how the software works on the editorial process flow screen. It outlines the basic steps for assessing relevance and plurality based on matrices that act as editorial planning tools. The system is designed to generate real-time production indicators based on information journalists provide regarding news values, topics and sources. These procedures, which are manual in the current version of the software, can be implemented with the help of artificial intelligence.

Figure 4. Operational overview of Qualijor, software that incorporates quality indicators into the management of the journalistic production process

Figure 4. Operational overview of Qualijor, software that incorporates quality indicators into the management of the journalistic production process

46The system incorporates fundamental aspects of structured journalism and DJDB but is at an early stage for professional application. Nevertheless, the concepts outlined in this article have been partially implemented. This technological possibility makes quality-orientated journalism feasible in the terms presented here, under significantly better conditions than in the past.

5. Final Considerations

47The current crisis in journalism’s business model and credibility underscores the critical need for quality. However, there is no consistent movement in the sector to provide journalism with a robust structure to promote and evaluate editorial quality. This article advocates two approaches focused on applied research and experimental development:

Creating a QMS based on an SAQJ. This requires reframing the concept of quality, not merely about evaluating individual products or processes, but conceiving quality as a system, the organisation’s efforts to promote and implement quality in its processes and products. It involves defining evaluation requirements and metrics to guide production, from the initial editorial concept to delivering the final product to the audience and society.

The second strategy involves leveraging the technological possibilities within DJDB and structured journalism. This aims to create content management and publishing software that enables the implementation of the QMS within journalistic organisations tailored to their specific professional practices.

48The proposals outlined here are still being determined. They certainly require many interventions to promote adjustments, improvements and the incorporation of new resources. These actions involve academic, professional and business sectors, among other interested parties.

49Both movements demand innovation. The current editorial model lacks metrics and rigorous methods for measuring the quality claimed by organisations, failing to uphold the trust placed by society in these professionals and companies. The first movement requires editorial innovation and the second calls for technological innovation. Their collaboration can yield a new journalistic intelligence essential for implementing the SAQJ and addressing the contemporary challenges in journalism.

To the Conselho Nacional de Desenvolvimento Científico e Tecnológico, a Brazilian research institution, funding the project Credibilidade Jornalística: Formulação de Indicadores de Fortalecimento do Jornalismo Para o Combate aos Ecossistemas de Desinformação (Journalistic Credibility Index: Formulating Indicators for Strengthening Journalism to Combat Disinformation Ecosystems; Process No. 403634/2021-0), of which this article is a partial result of one of the research strands.

Topo da página

Bibliografia

Anderson, C. W., Bell, E., & Shirky, C. (2012). Post-industrial journalism: Adapting to the present. Tow Center for Digital Journalism, Columbia University.

Anderson, P. J. (2014). Defining and measuring quality news journalism. In P. J. Anderson, G. Ogola, & M. Williams (Eds.), The future of quality news journalism: A cross-continental analysis (pp. 23–39). Routledge.

Associação Brasileira de Normas Técnicas. (2015a). ABNT NBR ISO 900: Sistemas de gestão da qualidade — Fundamentos e vocabulario. ABNT.

Associação Brasileira de Normas Técnicas. (2015b). ABNT NBR ISO 901: Sistemas de gestão da qualidade — Requisitos. ABNT.

Barbosa, S. (2008). Modelo jornalismo digital em base de dados (JDBD) em interação com a convergência jornalística. Textual & Visual Media, 1, 87–106.

Bertrand, C. J. (2002). O arsenal da democracia: Sistemas de responsabilização da mídia (M. L. Loureiro, Trans.). Editora da Universidade do Sagrado Coração. (Original work published 1999)

Carvalho, M. M. (2005). Histórico da gestão da qualidade. In M. M. Carvalho & E. P. Paladini (Eds.), Histórico da gestão da qualidade: Teoria e casos (pp. 1–24). Elsevier.

Chiavenato, I. (2007). Administração: Teoria, processo e prática (4th ed.). Elsevier.

Christofoletti, R. (2010). Indicadores da qualidade no jornalismo: Políticas, padrões e preocupações de jornais e revistas brasileiros. UNESCO.

Cristian, C. F., Glasser, T. L., McQuail, D., Nordesntreng, K., & White, R. A. (Eds.). (2009). Normative theories of the media: Journalism in democratic societies. University of Illinois Press.

Fengler, S., Eberwein, T., Mazzoleni, G., Porlezza, C., & Russ-Mohl, S. (Eds.). (2014). Journalists and media accountability: An international study of news people in the digital age. Peter Lang.

Franciscato, C. (2005). A fabricação do presente: Como o jornalismo reformulou a experiência do tempo nas sociedades ocidentais. Editora UFS/Fundação Oviêdo Teixeira.

Franciscato, C. (2006, November 8–10). Considerações metodológicas sobre a pesquisa aplicada em jornalismo [Conference presentation]. IV Encontro Nacional da Associação Brasileira dos Pesquisadores em Jornalismo, Porto Alegre, Brazil.

Freixa, P., Pérez-Montoro, M., & Codina, L. (2017). Interacción y visualización de datos en el periodismo estructurado. El Profesional de la Información, 26(6), 1076–1090. https://0-doi-org.catalogue.libraries.london.ac.uk/10.3145/epi.2017.nov.07

Gomes, W. (2009). Jornalismo, fatos e interesses. Ensaios de teoria do jornalismo. Insular.

Guerra, J. L. (2010). Sistema de gestão de qualidade aplicado ao jornalismo: Possibilidades e diretrizes. E-Compós, 13(3), 1–16. https://0-doi-org.catalogue.libraries.london.ac.uk/10.30962/ec.470

Guerra, J. L. (2016). Qualijor – Sistema de gestão da produção jornalística orientado para a qualidade editorial: Pesquisa aplicada e de desenvolvimento experimental em jornalismo. E-Compós, 19(3), 1–26. https://0-doi-org.catalogue.libraries.london.ac.uk/10.30962/ec.1291

Guerra, J. L. (2020). Ranking Q-Avalia da qualidade jornalística Brasil-Portugal 2018: Uma avaliação experimental. Estudos em Jornalismo e Mídia, 17(1), 54–75. https://doi.org/10.5007/1984-6924.2020v17n1p54

Guerra, J. L., & Feitoza, L. (2020). Relevância jornalística: Conceito, fundamentos e aplicação. LemD – Linguagem em (Dis)curso, 20(2), 401–419. https://0-doi-org.catalogue.libraries.london.ac.uk/10.1590/1982-4017-200210-10419

Gordillo, M. M., Valencia, O. B., & Cruz, I. R. (2013). El valor agregado periodístico, herramienta para el periodismo de calidad. In J. L. G. Mompart, J. F. G. Lozano, & D. P. Sampio (Eds.), La calidad periodística: Teorias, investigaciones y sugerencias profesionales (pp. 121–137). Universitat Autònoma de Barcelona/Universitat Pompeu Fabra.

Kovach, B., & Rosenstiel, T. (2001). The elements of journalism: What newspeople should know and the public should expect. Crown Publishers.

Lacy, S., & Rosenstiel, T. (2015). Defining and measuring quality journalism. Rutgers School of Communication and Information.

Lima Junior, W., & Oliveira, A. (2017). Jornalismo estruturado: Uso de metadados para enriquecimento de bases noticiosas na web. Revista FAMECOS, 24(3), Article ID25666. https://doi.org/10.15448/1980-3729.2017.3.25666

Machado, E., & Sant’Ana, J. (2014). Limitações metodológicas na pesquisa em jornalismo: Um estudo dos trabalhos apresentados no GT de jornalismo da COMPÓS (2000-2010). Pauta Geral, 1(1), 29–45. https://0-doi-org.catalogue.libraries.london.ac.uk/10.18661/2318-857X/pauta.geral.v1n1p26-42

McQuail, D. (2003). Responsibility and accountability conceptual distinctions. In D. McQuail (Ed.), Media accountability and freedom of publication (pp. 9–25). Oxford University Press.

McQuail, D. (2013). Principles of journalistic performance. In D. McQuail (Ed.), Journalism and society (pp. 27–39). SAGE.

Meditsch, E., & Moreno, C. A. de. (2004). Estudos em jornalismo. INTERCOM Revista Brasileira de Ciências da Comunicação, 27(2), 93–107. https://0-doi-org.catalogue.libraries.london.ac.uk/10.1590/rbcc.v27i2.1070

Meyer, P. (2007). Os jornais podem desaparecer? Como salvar o jornalismo na era da informação (P. de Cia, Trans.). Contexto. (Original work published 2004)

Mompart, J. L. G., & Sampio, D. P. (2013). El reto dela excelência: Indicadores para medir la calidad periodistica. In J. L. G. Mompart, J. F. G. Lozano, & D. P. Sampio (Eds.), La calidad periodística: Teorias, investigaciones y sugerencias profesionales (pp. 121–137). Universitat Autònoma de Barcelona/Universitat Pompeu Fabra.

Morales-Vargas, A., Pedraza-Jiménez, R., & Codina, L. (2022). Qualidade da web em mídia digital: Revisão bibliográfica sobre métodos e indicadores gerais de avaliação e atributos de confiança. Revista Latina de Comunicación Social, (80), 39–63. https://0-www-doi-org.catalogue.libraries.london.ac.uk/10.4185/RLCS-2022-1515

Newman, N., Fletcher, R., Robertson, C. T., Eddy, K., & Nielsen, R. (2022). Reuters institute digital news report 2022 (Vol. 2022). Reuters Institute for the Study of Journalism. https://reutersinstitute.politics.ox.ac.uk/sites/default/files/2022-06/Digital_News-Report_2022.pdf

News Guard. (2023). Rating process and criteria. Retrieved March 15, 2023, from https://www.newsguardtech.com/ratings/rating-process-criteria/

Odriozola-Chéné, J., Gutiérrez-Atala, F., Domínguez-Panamá, J. J., & Pérez-Arozamena, R. (2019). Las relaciones de las influencias en los procesos de producción informativa y sus efectos en la calidad periodística. Una visión desde Latinoamérica. Cuadernos.info, (44), 119–134. https://0-doi-org.catalogue.libraries.london.ac.uk/10.7764/ cdi.44.1297

Organisation for Economic Co-Operation and Development. (2015). Frascati manual 2015: Guidelines for collecting and reporting data on research and experimental development. OECD Publishing. https://doi.org/10.1787/9789264239012-en

Organisation for Economic Co-Operation and Development. (2018). Oslo manual 2018: Guidelines for collecting, reporting and using data on innovation. OECD Publishing. https://doi.org/10.1787/9789264304604-en

Palacios, M. (2011). Ferramentas para análise de qualidade no ciberjornalismo. Volume 1: Modelos. LabCom Books.

Paladini, E. P. (2005). Perspectiva estratégica da qualidade. In M. M. Carvalho & E. P. Paladini (Eds.), Histórico da gestão da qualidade: Teoria e casos (pp. 25–84). Elsevier.

Pinto, M., & Marinho, S. (2003, April 10–11). A qualidade em jornalismo: Problematização e operacionalização do conceito [Conference presentation]. I Congresso Luso-Brasileiro de Estudos Jornalísticos e ao II Congresso Luso-Galego de Estudos Jornalísticos, Porto, Portugal. https://0-hdl-handle-net.catalogue.libraries.london.ac.uk/1822/997

Reporters Without Borders. (2021). Journalim Trust Initiative. Retrieved March 15, 2023, from https://rsf.org/en/journalism-trust-initiative

Rosengren, K. E., Carlsson, M., & Tagerud, Y. (1996). Quality in programming: Views from the North. In S. Ishikawa (Ed.), Quality assessment of television (pp. 49–60). University of Luton Press.

Rothberg, D., & Garrido, B. A. (2019). Jornalismo, gestão da qualidade e regulação: Estudo comparado de 42 corporações de mídia. Estudos em Jornalismo e Mídia, 16(1), 85–97. https://doi.org/10.5007/1984-6924.2019v16n2p85

Salaverría, R. (2003). Convergencia de los medios. Chasqui, 81, 32–39.

Santos, E., & Guazina, L. (2020). Qualidade no jornalismo: Percursos estrangeiros, problemas brasileiros.

Estudos em Jornalismo e Mídia, 17(1), 32–47. https://0-doi-org.catalogue.libraries.london.ac.uk/10.5007/1984-6924.2020v17n1p32 Schwingel, C. (2012). Ciberjornalismo. Paulinas.

Slack, N., Chambers, S., & Johnston, R. (2007). Administração da produção (M. T. Corrêa, F. Alher, & H. L. Corrêa, Trans.). Atlas. (Original work published 1997)

Strömbäck, J. (2005). In search of a standard: Four models of democracy and their normative implications for journalism. Journalism Studies, 6(3), 331–345. https://0-doi-org.catalogue.libraries.london.ac.uk/10.1080/14616700500131950

Swanepoel, T. (2012). A comprehensive quality management model for community newspapers [Doctoral dissertation, North-West University].

Torre, L., & Téramo, M. T. (2007). Investigación sobre calidad periodística em la Argentina. In A. A. Suárez (Ed.), Periodismo de calidade: Debates y desafios (pp. 39–62). La Crujia; Fopea.

Träsel, M., Lisboa, S., & Reis, G. (2018). Indicadores de credibilidade no jornalismo: Uma análise dos produtores de conteúdo político brasileiros. In Anais do XXVII Encontro Anual da Compós (pp. 1–19). Pontifícia Universidade Católica de Minas Gerais.

The Trust Project. (2023). About us. Retrieved March 15, 2023, from https://thetrustproject.org/about/

Vehkoo, J. (2009/2010). What is quality journalism and how it can be saved. Reuters Institute for the Study of Journalism.

Topo da página

Notas

1 The term “journalistic institution” refers to the collective knowledge and interested parties (professionals, organisations and society) interlinked to form the entity recognised as “journalism”. A “journalistic organisation” represents a specific group of individuals and resources actively applying professional expertise to practice journalism.

2 To explore further insights and references on applied research in journalism, consider works by Meditsch and Moreno (2004), Franciscato (2006), and Machado and Sant’Ana (2014). The description of the concept and method of applied research in this article echoes fundamental aspects and passages developed in other articles, such as Guerra (2016).

3 Due to this text’s scope, detailed development of these primary requirement concepts is not possible. For the concept of “truth”, refer to Gomes (2009); for “recentness”, see Franciscato (2005); for “relevance”, refer to Guerra and Feitoza (2020).

Topo da página

Índice das ilustrações

Título Figure 1. Concise diagram outlining the interactions, interested parties, and organisational dynamics intended to form a quality management system applied to journalistic organisations
Legenda Note. Created by the author. An earlier, slightly different version is available in Guerra (2020)
URL http://0-journals-openedition-org.catalogue.libraries.london.ac.uk/cs/docannexe/image/10324/img-1.jpg
Ficheiro image/jpeg, 172k
Título Figure 2. Alignment of requirements to integrate the production process
URL http://0-journals-openedition-org.catalogue.libraries.london.ac.uk/cs/docannexe/image/10324/img-2.png
Ficheiro image/png, 22k
Título Figure 3. Structured requirements for a quality management system generate parameters for a computerised editorial production management system focused on quality
URL http://0-journals-openedition-org.catalogue.libraries.london.ac.uk/cs/docannexe/image/10324/img-3.png
Ficheiro image/png, 44k
Título Figure 4. Operational overview of Qualijor, software that incorporates quality indicators into the management of the journalistic production process
URL http://0-journals-openedition-org.catalogue.libraries.london.ac.uk/cs/docannexe/image/10324/img-4.jpg
Ficheiro image/jpeg, 200k
Topo da página

Para citar este artigo

Referência do documento impresso

Josenildo Luiz Guerra, «The Strategic Approach to Quality in Journalism: Innovation, Technology and Applied Research»Comunicação e sociedade, 44 | 2023, 1-18.

Referência eletrónica

Josenildo Luiz Guerra, «The Strategic Approach to Quality in Journalism: Innovation, Technology and Applied Research»Comunicação e sociedade [Online], 44 | 2023, posto online no dia 07 dezembro 2023, consultado o 21 junho 2024. URL: http://0-journals-openedition-org.catalogue.libraries.london.ac.uk/cs/10324

Topo da página

Autor

Josenildo Luiz Guerra

Josenildo Luiz Guerra is a full professor at the Federal University of Sergipe, contributing to the Journalism course and the Postgraduate Programme in Communication. He holds a PhD in Communication and Contemporary Culture from the Universidade Federal da Bahia (2003) and pursued post-doctoral studies at the University of Minho (Portugal; 2017/2018). He is one of the leaders of the Journalism Studies Laboratory and the coordinator of Qualijor (Program of Research Quality, Innovation and Applied Technology in Journalism), affiliated with the Rede Nacional de Observatórios da Imprensa. Guerra specialises in applied research and experimental development, devising technical and technological solutions to enhance journalistic practices and quality assessment. He is currently part of the Índice de Credibilidade Jornalística project, collaborating with researchers from five Brazilian universities.
ORCID: https://orcid.org/0000-0001-5242-863X
Email: jguerra@academico.ufs.br
Address: Departamento de Comunicação Social, Universidade Federal de Sergipe. Cidade Univ. Prof. José Aloísio de Campos, Av. Marcelo Deda Chagas, s/n, Bairro Rosa Elze, São Cristóvão, Sergipe, Brasil. CEP 49107-230

Topo da página

Direitos de autor

CC-BY-4.0

Apenas o texto pode ser utilizado sob licença CC BY 4.0. Outros elementos (ilustrações, anexos importados) são "Todos os direitos reservados", à exceção de indicação em contrário.

Topo da página
Pesquisar OpenEdition Search

Você sera redirecionado para OpenEdition Search